commandline data analytics

Here are some patterns that were important for most infrastructure functions - infrastructure operations, software development, or even large scale infrastructure changes, migrations and predicting future states.

The slice: "| grep | project"
The map-reduce pipeline: "| transform | project | sort |uniq -c|sort -nr"
The pivot: "|project | pivot-and-aggregate"

The output of these patterns are typically fed into the execution pipeline.

These patterns are essential because they direct, focus and scale the primary output beyond what is possible by just being an expert.

There is nothing magical about any of those. But as a considered habit, an involuntary muscle memory, they can bring enormous leverage and time line compression to any complex piece of work.

There is one pattern that is missing from this: classification/machine learning filter.

We need an "ml-filter" to make the future architect/webops engineer's end result to become a fully closed loop control system.

Comments

Why PI is not 4, math is great, and other mysteries.

The other day, I found myself with an interesting problem of approximating a circle with the enclosing square which seems to prove pi = 4. The paradox was forwarded by a most interesting puzzle collector, Surajit Basu, a friend and life long inspiration. See Sonata for Unaccompanied Tortoise for why! Here is the offending paradox: This is an example of how counterintuitive questions can be answered with a little calculus. The key is to realize that no matter how closely we approximate the circle, the orthogonal lines of the approximation formed by inverting the square corners will never actually be tangential to the circle. Note carefully that as you get closer to 90 degrees, the horizontal line is much longer than the vertical. Same goes with the approximation at 0 and 180 - the vertical line is much larger than the horizontal component. If we take a quadrant of the circle - let's say the top left quadrant, moving counter clockwise from to

Architecture, Engineering, Operations - 1

The world has infinitely more stuff to be "done" nowadays. At least in the sense of building/running an institution that uses technology, there are many roles that are involved in making things work. The world of IT and technology in general makes the speed and variety possible. We now have a platform of IT that is globally scale-able if we can put some new thinking to the old problems of "getting things done". There are great organizations that do this well, and they use modern IT principles to achieve this. Fundamental to engineering a modern IT (or infrastructure organization) are the three roles of Architecture, Engineering and Operations. Some would say Architecture is encoded Engineering-history, but for now, we will keep them separate. The popular definitions for these roles are about output delivered or the domain of discourse. The personality drives that determine the actual performance are not discussed, as far as I can see, in a holistic fashion i

Ambition vs. Fear.

Most important things in life don't come to us. Nor do we get them by seeking/wanting them. It comes from letting go of the unimportant stuff. The hardest part is letting go of the tendency to take the world as is. This is a habit of our past successes. But success is not a destination, it is a STOP sign. You stop, wait, and move on. Too often, we are paralyzed by success into the fear of the new. We stall on the road to a new life. We need to break our inertia and move. Our thoughts and thought habits are hard to break. But that is where we have to spend the most energy. Thoughts are always competing strands - of worries of the past and anxieties for the future. For some of us, they are cleanly separated into rivers that nurture every place they travel. For most, they are like the torrents and trickles -- competing, rushing somewhere, stopping completely elsewhere, always mixing, morphing, competing, winning, losing. Our thoughts are the potential difference between the t

InfraStruct

Search This Blog