Pippo's blog

it is all about software development

Keep automating

When you start automating routines that you do more than a couple times, you tend to want to automate everything. Why? I think because it is challenging and funny. I am finding many opportunities to automate my routines these days and I am trying to keep the automation as part of my bash environment so I can reuse them as much as possible and have easy access to what I already did.

One of the challenges is to write code that you already know how to do in a different language like Ruby but to keep it in bash you have no clue.

The mix of languages bash can provide you with good support can enhance this experience a little, but don’t do too much outside of the bash world, otherwise you loose the benefits of doing so as functions to your terminal.

What I want to show you is this snippet where I can parse a JSON file and get the value I want directly from the terminal and being able to pipe it to another functions.

I was calling this google maps API, to retrieve latitude and lonngitude for an address, just like this:

1
2
url='maps.google.com/maps/api/geocode/json?sensor=false&address=Catedral+Metropolitana+de+Porto+Alegre+-+Rua+Esp%C3%ADrito+Santo,+Porto+Alegre,+Brazil&hl=en&ie=UTF8&sll=40.697299,-73.486176&sspn=0.754849,1.448822&oq=Catedral+Metropolitana+de+Porto+Alegre&hq=Catedral+Metropolitana+de+Porto+Alegre+-&hnear=R.+Esp%C3%ADrito+Santo+-+Centro,+Porto+Alegre+-+Rio+Grande+do+Sul,+90010-370,+Brazil'
http GET $url

Oh, and I use HTTPie as my default command line REST client ;-)

What this API return is a not-so-small JSON.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
{
   "results" : [
      {
         "address_components" : [
            {
               "long_name" : "Catedral Metropolitana de Porto Alegre",
               "short_name" : "Catedral Metropolitana de Porto Alegre",
               "types" : [ "point_of_interest", "establishment" ]
            },
            {
               "long_name" : "55",
               "short_name" : "55",
               "types" : [ "street_number" ]
            },
            {
               "long_name" : "Rua Espírito Santo",
               "short_name" : "R. Espírito Santo",
               "types" : [ "route" ]
            },
            {
               "long_name" : "Centro Historico",
               "short_name" : "Centro Historico",
               "types" : [ "sublocality", "political" ]
            },
            {
               "long_name" : "Porto Alegre",
               "short_name" : "Porto Alegre",
               "types" : [ "locality", "political" ]
            },
            {
               "long_name" : "Rio Grande do Sul",
               "short_name" : "RS",
               "types" : [ "administrative_area_level_1", "political" ]
            },
            {
               "long_name" : "Brazil",
               "short_name" : "BR",
               "types" : [ "country", "political" ]
            },
            {
               "long_name" : "90010-370",
               "short_name" : "90010-370",
               "types" : [ "postal_code" ]
            }
         ],
         "formatted_address" : "Catedral Metropolitana de Porto Alegre - Rua Espírito Santo, 55 - Central, Porto Alegre - Rio Grande do Sul, 90010-370, Brazil",
         "geometry" : {
            "location" : {
               "lat" : -30.0335439,
               "lng" : -51.2295866
            },
            "location_type" : "APPROXIMATE",
            "viewport" : {
               "northeast" : {
                  "lat" : -30.0238836,
                  "lng" : -51.2135792
               },
               "southwest" : {
                  "lat" : -30.0432033,
                  "lng" : -51.245594
               }
            }
         },
         "types" : [ "place_of_worship", "point_of_interest", "establishment" ]
      }
   ],
   "status" : "OK"
}

… I just wanted latitude and longitude for the address on the url, and I wanted that information for a bunch of different addresses. Basicaly, I needed to navigate the JSON: first object on results array, than geometry node and get latitude/longitude fields from location subnode.

I came up with this function that allows me to do just that passing the nodes I want to navigate:

1
2
3
4
5
6
7
8
9
10
parse_json() {
  attrs=""
  for i in $*;
  do
    local regex='^[0-9]+$'
    local param=$([[ ${i} =~ ${regex} ]] && echo  ${i} || echo "'${i}'" )
    attrs=$attrs"[${param}]"
  done
  ruby -e "require 'json'; parsed = JSON.parse(STDIN.read)$attrs; puts parsed.to_json";
}

Having so I can store the JSON response in a variable to hit the API only once, and than reuse the JSON with diferent parsing arguments:

1
2
response=$(http GET $url)
echo $response | parse_json results 0 geometry

and still have a json result

1
{"location":{"lat":-30.0335439,"lng":-51.2295866},"location_type":"APPROXIMATE","viewport":{"northeast":{"lat":-30.0238836,"lng":-51.2135792},"southwest":{"lat":-30.0432033,"lng":-51.245594}}}

or narrow it down to location

1
echo $response | parse_json results 0 geometry location

almost there

1
{"lat":-30.0335439,"lng":-51.2295866}

and directly to one of the attributes we want

1
echo $response | parse_json results 0 geometry location lat

result

1
-30.0335439

Cool, hein?!

Remember I would like to be able to pipe the result to another command? So running the geometry filter again, with a one more command in the pipe:

1
echo $response | parse_json results 0 geometry | pretty_json

now with a pretty formatted json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
    "location": {
        "lat": -30.0335439,
        "lng": -51.2295866
    },
    "location_type": "APPROXIMATE",
    "viewport": {
        "northeast": {
            "lat": -30.0238836,
            "lng": -51.2135792
        },
        "southwest": {
            "lat": -30.0432033,
            "lng": -51.245594
        }
    }
}

pretty_json is an alias I have, which is basicly a call to a python function:

1
2
$ type pretty_json
pretty_json is aliased to `python -mjson.tool'

Being able to write small functions and glue them together in a pipe or reusing them in a loop can save me many manual work and also help me to learn bash more and more! I hope you do this kind of thing as well!

Comments