Friday, 28 April 2017

HTTP request headers


Check request header with cURL (see URL)
- cURL  is  a  tool  to transfer data from or to a server, using one of the supported protocols


$ curl -v -XGET 172.24.14.158
* Rebuilt URL to: 172.24.14.158/
*   Trying 172.24.14.158...
* Connected to 172.24.14.158 (172.24.14.158) port 80 (#0)
> GET / HTTP/1.1
> Host: 172.24.14.158
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 200 
< Content-Type: text/html;charset=ISO-8859-1
< Date: Sun, 30 Apr 2017 06:29:31 GMT
< Set-Cookie: JSESSIONID=C565042EC3A26A25D4ECF6787B54CE76;path=/;HttpOnly
< Content-Length: 14
< Connection: keep-alive
< 
some.her.data
* Connection #0 to host 172.24.14.158 left intact

Check request header with browser



Sunday, 26 March 2017

Playing with java date format


TimeZone examples

On a machine with default timezone as US/Pacific

scala> import java.text.SimpleDateFormat
scala> import java.util.TimeZone
scala> import java.util.Date

scala> TimeZone.getDefault()
res26: java.util.TimeZone = sun.util.calendar.ZoneInfo[id="US/Pacific",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=US/Pacific,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]


scala> val dateFormat = new SimpleDateFormat("yyyy-MM-dd")
dateFormat: java.text.SimpleDateFormat = java.text.SimpleDateFormat@f67a0200

scala> dateFormat.setTimeZone(TimeZone.getTimeZone("US/Pacific"))

scala> val date = dateFormat.format(new Date(2017-1900, 10-1, 28)) //because new Date(2017-1900, 10-1, 28) takes default timezone
warning: there was one deprecation warning; re-run with -deprecation for details
date: String = 2017-10-28

scala> dateFormat.parse(date).getTime
res1: Long = 1509174000000

On a machine with default timezone as America/Los_Angeles,

scala> TimeZone.getDefault()
res11: java.util.TimeZone = sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]

scala> val dateFormat = new SimpleDateFormat("yyyy-MM-dd")
dateFormat: java.text.SimpleDateFormat = java.text.SimpleDateFormat@f67a0200

scala> dateFormat.setTimeZone(TimeZone.getTimeZone("America/Los_Angeles"))

scala> val date = dateFormat.format(new Date(2017-1900, 10-1, 28))
<console>:18: warning: constructor Date in class Date is deprecated: see corresponding Javadoc for more information.
       val date = dateFormat.format(new Date(2017-1900, 10-1, 28))
                                    ^
date: String = 2017-10-28

scala> dateFormat.parse(date).getTime
res13: Long = 1509174000000

On a machine with default timezone as Etc/UTC

scala> import java.text.SimpleDateFormat
scala> import java.util.TimeZone
scala> import java.util.Date

scala> TimeZone.getDefault()
res9: java.util.TimeZone = sun.util.calendar.ZoneInfo[id="Etc/UTC",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]

scala> val dateFormat = new SimpleDateFormat("yyyy-MM-dd")
dateFormat: java.text.SimpleDateFormat = java.text.SimpleDateFormat@f67a0200

scala> dateFormat.setTimeZone(TimeZone.getTimeZone("US/Pacific"))

scala>  val date = dateFormat.format(new Date(2017-1900, 10-1, 28))
warning: there was one deprecation warning; re-run with -deprecation for details
date: String = 2017-10-27

scala>  dateFormat.parse(date).getTime
res2: Long = 1509087600000

date formats

scala> import java.util.Calendar
import java.util.Calendar

scala> val today = Calendar.getInstance().getTime()
today: java.util.Date = Sun Mar 26 00:58:12 PDT 2017

scala> import java.text.SimpleDateFormat
import java.text.SimpleDateFormat

scala> val hourMinuteSecs = new SimpleDateFormat("kkmmss").format(today)
hourMinuteSecs: String = 245812

scala> val hourMinuteSecs = new SimpleDateFormat("hhmmss").format(today)
hourMinuteSecs: String = 125812

scala> val hourMinuteSecs = new SimpleDateFormat("HHmmss").format(today)
hourMinuteSecs: String = 005812

scala> val hourMinuteSecs = new SimpleDateFormat("HHmmss a").format(today)
hourMinuteSecs: String = 005812 AM

Tuesday, 14 March 2017

scala value class vs ref class

Scala Value class vs Ref class

Int (including FloatDoubleChar etc) is a value class in scala, which is totally different thing than Null trait when you do val x: Int = null.
final abstract class Int() extends scala.AnyVal {}
What is a value class?
Value class is a class whose instances are not represented as
objects by the underlying host system. All value classes inherit from
class AnyVal.
You can try your creating your own value class,
scala> case class ValString(str: String) extends AnyVal
defined class ValString

scala> val string: ValString = null
<console>:13: error: type mismatch;
 found   : Null(null)
 required: ValString
       val string: ValString = null
                               ^
value class needs to have some value or none. Thats why its recommended to use Option[ValString]
On the other hand, String is AnyRef. Actually AnyRef corresponds to java.lang.Object.
example,
scala> class RefString(str: String) extends AnyRef
defined class RefString

scala> val refString : RefString = null
refString: RefString = null

Saturday, 14 January 2017

hash functions/ cryptographic hash functions


hash functions/ cryptographic hash functions

a mathematical algorithm that maps data of arbitrary size to a bit string of a fixed size (a hash function)
which is designed to also be a one-way function

secure hash(s-ha)

SHA-1 produces a 160-bit (20-byte) hash value known as a message digest.
A SHA-1 hash value is typically rendered as a hexadecimal number, 40 digits long = 40 digits/2 bytes.
$ echo "1" | shasum -a 1
e5fa 44f2 b31c 1fb5 53b6 021e 7360 d07d 5d91 ff5e  -

$ echo "11" | shasum -a 1
dd71 038f 3463 f511 ee74 03db cbc8 7195 302d 891c  -

$ printf 'dd71038f3463f511ee7403dbcbc87195302d891c' | wc -c
      40

$ shasum downloadData.log -a 1
f500ddd45af385b3bbdffdc3457701bf5b9a37a1  downloadData.log

Java api

scala> val hash = java.security.MessageDigest.getInstance("SHA-1").digest("1".getBytes())
x: Array[Byte] = Array(53, 106, 25, 43, 121, 19, -80, 76, 84, 87, 77, 24, -62, -115, 70, -26, 57, 84, 40, -85)

scala> new String(hash)
res5: String = 5j?+y?LTWM?F�9T(�

scala> new BigInteger(java.security.MessageDigest.getInstance("SHA-1").digest("1".getBytes()))
res8: java.math.BigInteger = 304942582444936629325699363757435820077590259883

scala> new BigInteger(java.security.MessageDigest.getInstance("SHA-1").digest("1".getBytes())).toString(16)
res9: String = 356a192b7913b04c54574d18c28d46e6395428ab

Merkle–Damg√•rd 5

The MD5 algorithm is a widely used hash function producing a 128-bit hash value.
Used for checksum match against a file, but is very vulnerable with collisions within seconds.

Following is BSD example, Linux has md5sum.

$ echo -n "1" | md5
c4ca4238a0b923820dcc509a6f75849b

$ echo "1" | md5 # don't do this, as includes \n
b026324c 6904b2a9 cb4b88d6 d61c81d1 ##is a [hex number, compressed to base16](http://stackoverflow.com/q/43556742/432903)

#hash bytes
printf "%s" "c4ca4238a0b923820dcc509a6f75849b" | wc -c
      32
# hash bits = (32/2=16bytes)*8 = 128bits

## another way
$ md5 <<<"1"
b026324c6904b2a9cb4b88d6d61c81d1

$ md5 build.sbt 
MD5 (build.sbt) = d10c 6aff 431a 61c5 b3bd 1a03 8519 900c


Java MD5 API

scala> val hash = MessageDigest.getInstance("MD5").digest("1".getBytes("UTF-8"))
hash: Array[Byte] = Array(-60, -54, 66, 56, -96, -71, 35, -126, 13, -52, 80, -102, 111, 117, -124, -101)

scala> val hash = MessageDigest.getInstance("MD5").digest("1".getBytes("UTF-8")).length
hash: Int = 16

scala> import java.math.BigInteger
import java.math.BigInteger

scala> new BigInteger(1, MessageDigest.getInstance("MD5").digest("1".getBytes("UTF-8")))
res5: java.math.BigInteger = 261578874264819908609102035485573088411 //length 39

scala> new BigInteger(1, MessageDigest.getInstance("MD5").digest("1".getBytes("UTF-8"))).toString(16)
res6: String = c4ca4238a0b923820dcc509a6f75849b //length 32

scala> String.format("%032x", new BigInteger(1, MessageDigest.getInstance("MD5").digest("1".getBytes("UTF-8"))))
res7: String = c4ca4238a0b923820dcc509a6f75849b


Usage

1) data distribution/partitioning
http://docs.aws.amazon.com/streams/latest/dev/key-concepts.html

2) How do I ensure data integrity of objects uploaded to or downloaded from Amazon S3? - eTag/ Content-MD5

References




Sunday, 4 December 2016

create a OS managed service using systemd

  1. UNIX System V OS/ BSD OS init systems,
  2. upstart, 2006, an event-based replacement for the traditional init daemon upstart.
    The recent version of ubuntu uses systemd, as far as I remember previous versions of ubuntu use upstart.
  3. runit, 2004
  4. launchd in MacOS world.
PS. There's lot of hate against systemd, it being against Unix philosophy that goes as;
Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.
Here's a thread which might be useful to dig.

Fun time

I'm creating a streaming service in this example, which will simply write an event to a file continuously. And, if I restart my machine the service will start emitting events to a file right away.
Also I can start or stop service using systemctl start/stop serviceName. I used to use nohup <command> & and to kill it I had to use get the PID and then kill it. I can get rid of all that crap with systemD service.
STEP 1 - create streaming.service at /etc/systemd/system/multi-user.target.wants/streaming.service, also at /etc/systemd/system/streaming.service.
[Unit]                                                                                                                                           
Description=Streaming pipeline                                                                      

[Service] 
Type=forking                                                                         
ExecStart=/usr/local/bin/streaming.sh
TimeoutSec=infinity
Restart=always

[Install]
WantedBy=multi-user.target
STEP 2 - create a bash script at /usr/local/bin/streaming.sh and give current user permission to it. Don't forget #!/bin/bash for the bash scripts.
#!/bin/bash
while true
 do
    echo "Streaming an event at `date`" >> /var/log/streaming.log
 done
chmod 777 /usr/local/bin/streaming.sh
STEP 3 - reload systemd daemon so that it loads streaming.service and enable the service
systemctl daemon-reload
systemctl enable streaming.service
STEP 4 - see the streaming job status
systemctl status streaming.service
I can use journalctl _PID=?? to debug if the service is not working. eg.
journalctl _PID=5871
-- Logs begin at Mon 2016-11-21 16:36:41 PST, end at Tue 2016-12-06 14:34:56 PST. --
Nov 26 04:17:52 y0319t10971 sshd[5871]: Connection closed by 10.16.132.191 [preauth]
Dec 06 14:33:15 y0319t10971 systemd[5871]: Failed at step EXEC spawning /usr/local/bin/streaming.sh: Exec format error
or also can use following command,
root@y0319t10971:~# journalctl -fu streaming.service
-- Logs begin at Mon 2016-11-21 16:36:41 PST. --
Dec 06 14:33:16 y0319t10971 systemd[1]: streaming.service: main process exited, code=exited, status=203/EXEC
Dec 06 14:33:16 y0319t10971 systemd[1]: Unit streaming.service entered failed state.
Dec 06 14:33:16 y0319t10971 systemd[1]: streaming.service failed.
Dec 06 14:34:28 y0319t10971 systemd[1]: Started Streaming pipeline.
Dec 06 14:34:28 y0319t10971 systemd[1]: Starting Streaming pipeline...
Dec 06 14:34:28 y0319t10971 systemd[1]: streaming.service: main process exited, code=exited, status=203/EXEC
Dec 06 14:34:28 y0319t10971 systemd[1]: Unit streaming.service entered failed state.
Dec 06 14:34:28 y0319t10971 systemd[1]: streaming.service failed.
Dec 06 14:44:58 y0319t10971 systemd[1]: Started Streaming pipeline.
Dec 06 14:44:58 y0319t10971 systemd[1]: Starting Streaming pipeline...
STEP 5 - see the events produced by the streaming.service
tail -f /var/log/streaming.log 
Streaming an event at Fri Dec  2 11:27:21 PST 2016
Streaming an event at Fri Dec  2 11:27:21 PST 2016
Streaming an event at Fri Dec  2 11:27:21 PST 2016
Streaming an event at Fri Dec  2 11:27:21 PST 2016
Streaming an event at Fri Dec  2 11:27:21 PST 2016
Streaming an event at Fri Dec  2 11:27:21 PST 2016
Streaming an event at Fri Dec  2 11:27:21 PST 2016
Streaming an event at Fri Dec  2 11:27:21 PST 2016
Streaming an event at Fri Dec  2 11:27:21 PST 2016
Streaming an event at Fri Dec  2 11:27:21 PST 2016
PS. Dont forget to stop the service, otherwise your Storage will be full in few days :)
ll /var/log/streaming.log --block-size=GB
-rw-r--r-- 1 root root 3GB Dec  6 13:51 /var/log/streaming.log
Also, the service would be running right after I reboot the machine.