Skip to main content

commandline data analytics

Here are some patterns that were important for most  infrastructure functions - infrastructure operations,  software development, or even large scale infrastructure changes, migrations and predicting future states.

The slice:   "| grep | project"
The  map-reduce pipeline:   "| transform | project | sort |uniq -c|sort -nr"
The pivot:   "|project | pivot-and-aggregate"

The output of these patterns are typically fed into the execution pipeline.

These patterns are essential because they direct, focus and scale the primary output beyond what is possible by just being an expert.

There is nothing magical about any of those. But as a considered habit, an involuntary muscle memory, they can bring  enormous leverage and time line compression to any complex piece of work.

There is one pattern that is missing from this: classification/machine learning filter.

We need an  "ml-filter" to  make the future architect/webops engineer's end result to become a fully closed loop control system.



 

Popular posts from this blog

Why PI is not 4, math is great, and other mysteries.

The other day, I found myself with an interesting problem of approximating a circle with the enclosing square which seems to prove pi = 4.

The paradox was forwarded by a most interesting puzzle collector, Surajit Basu, a friend and life long inspiration. See Sonata for Unaccompanied Tortoise for why!



Here is the offending paradox:

























This is an example of how counterintuitive questions can be answered with a little calculus.

The key is to realize that no matter how closely we approximate the circle, the orthogonal lines of the approximation formed by inverting the square corners will never actually be tangential to the circle.

Note carefully that as you get closer to 90 degrees, the horizontal line is much longer than the vertical. Same goes with the approximation at 0 and 180 - the vertical line is much larger than the horizontal component.

If we take a quadrant of the circle - let's say the top left quadrant, moving counter clockwise from top to left -  we can imagine that each inf…

Ambition vs. Fear.

Most important things in life don't come to us. Nor do we get them by seeking/wanting them. It comes from letting go of the unimportant stuff.

The hardest part is letting go of the tendency to take the world as is. This is a habit of our past successes.

But success is not a destination, it is a STOP sign. You stop, wait, and move on. Too often, we are paralyzed by success into the fear of the new. We stall on the road to a new life. We need to break our inertia and move.

Our thoughts and thought habits are hard to break. But that is where we have to spend the most energy. Thoughts are always competing strands  - of worries of the past and anxieties for the future. For some of us, they are cleanly separated into rivers that nurture every place they travel. For most, they are like the torrents and trickles -- competing, rushing somewhere, stopping completely elsewhere, always mixing, morphing, competing, winning, losing.

Our thoughts are the potential difference between the two pole…

Meta leadership

Leadership The word leadership has a rather large spread and reach. It can mean a diverse set of behaviors. It can explain a wide variety of positive outcomes. It can effect a wide scope of results - from moving individuals within one's sphere of direct interaction, to moving an entire world far beyond one's direct influence. 
At its core, leadership is an attitude for influencing and executing transformation. It is a skill that results in exponential outcomes with a linear quantum of resource: Oneself. In this process, you leverage many sources both within, and outside, clear lines of control - resources, people,  organizations, domains of expanding knowledge, mental models, systems for doing things.

In today's world, leadership is probably the most important skill at every level of activity one wants to be successful at. This is simply because we are now connected to potentially billions of people within the palm of our hands, or within the reach of a keyboard.

Countles…