Vijay Tatkar will be talking about developing on Solaris next week Tuesday at 9am PST.
So we should have some news later this week on a minor code release with some helpful features in it. Can't say more until it comes out, but watch my blog this week.
In the meantime.... I have always been the grill-master at our camps with friends and family. My boys and I camp about 25-30 times a year. As much as I enjoy grilling, I was woefully behind in my smoking/BBQ skills. The difference being that grilling is cooking fast over high heat (think burgers, steak, and most seafood), and real BBQ involves smoke and slow-cooking over hours. Smoking is better for ribs, chicken, brisket and tri-tip. So I went to a real BBQ day-long class, got a small beginner's smoker, and now I'm smoking meat a lot more. Here's a pic of my last tri-tip in the smoker. Homemade rub and sauce cost just pennies compared to store-bought, and the meat is cheap at Costco. This may have been the best tri-tip I've ever made. Great smoke ring and flavor in only 1.5 hours. I was trying to tie this into the ZFSSA, but I just can't, so I stopped caring and now just showing off my new BBQ skills. Ha ha. Enjoy.
I work with my federal government and US DoD customers, and I'm frequently asked whether Oracle product X is IPv6:
This is because the Federal Acquisition Regulations require that the government purchase IPv6 compliant products.
Unless the agency Chief Information Officer waives the requirement, when acquiring information technology using Internet Protocol, the requirements documents must include reference to the appropriate technical capabilities defined in the USGv6 Profile (NIST Special Publication 500-267) and the corresponding declarations of conformance defined in the USGv6 Test Program.
Let's examine each of these adjectives one by one.
At the DISA mission partner conference this week, I attended a presentation by the DoD IPv6 Transition Office. The slides are available online. I asked the speaker if there is an "accepted" way of advertising IPv6 compliance and received no answer. He has promised to get back to me, however.
Oracle is a very large company with an extensive production encompassing storage, servers, thin clients, databases, middleware and application. I have found no single resource documenting the IPv6 status of every product. I can tell you, however, that Solaris 10 and Solaris 11 have successfully completed the USGv6 testing by the UNH Interoperability IPv6 test facility and the results are posted at their site.
As for Oracle Linux, it is fully compatible with Red Hat Linux 5 and 6 which has already been tested by UNH as well.
Note: I intended to provide additional references on USGv6 profiles and "Suppliers Declaration of Conformance" but the NIST web page seems to be in disrepair and the pages are not available.
Part of the value of dynamic and interpreted environments is that they handle the complexities of dynamic memory allocation. In particular, one needn’t explicitly free memory that is no longer in use: objects that are no longer referenceable are found automatically and destroyed via garbage collection. While garbage collection simplifies the program’s relationship with memory, it not mean the end of all memory-based pathologies: if an application retains a reference to an object that is ultimately rooted in a global scope, that object won’t be considered garbage and the memory associated with it will not be returned to the system. If enough such objects build up, allocation will ultimately fail (memory is, after all, finite) and the program will (usually) fail along with it. While this is not strictly — from the native code perspective, anyway — a memory leak (the application has not leaked memory so much as neglected to unreference a semantically unused object), the effect is nonetheless the same and the same nomenclature is used.
While all garbage collected environments create the potential to create such leaks, it can be particularly easy in JavaScript: closures create implicit references to variables within their scopes — references that might not be immediately obvious to the programmer. And node.js adds a new dimension of peril with its strictly asynchronous interface with the system: if backpressure from slow upstream services (I/O, networking, database services, etc.) isn’t carefully passed to downstream consumers, memory will begin to fill with the intermediate state. (That is, what one gains in concurrency of operations one may pay for in memory.) And of course, node.js is on the server — where the long-running nature of services means that the effect of a memory leak is much more likely to be felt and to affect service. Take all of these together, and you can easily see why virtually anyone who has stood up node.js in production will identify memory leaks as their most significant open issue.
The state of the art for node.js memory leak detection — as concisely described by Felix in his node memory leak tutorial — is to use the v8-profiler and Danny Coates’ node-inspector. While this method is a lot better than nothing, it’s very much oriented to the developer in development. This is a debilitating shortcoming because memory leaks are often observed only after days (hours, if you’re unlucky) of running in production. At Joyent, we have long wanted to tackle the problem of providing the necessary tooling to identify node.js memory leaks in production. When we are so lucky as to have these in native code (that is, node.js add-ons), we can use libumem and ::findleaks — a technology that we have used to find thousands of production memory leaks over the years. But this is (unfortunately) not the common case: the common case is a leak in JavaScript — be it node.js or application code — for which our existing production techniques are useless.
So for node.js leaks in production, one has been left with essentially a Gedanken experiment: consider your load and peruse source code looking for a leak. Given how diabolical these leaks can be, it’s amazing to me that anyone ever finds these leaks. (And it’s less surprising that they take an excruciating amount of effort over a long period of time.) That’s been the (sad) state of things for quite some time, but recently, this issue boiled over for us: Voxer, a Joyent customer and long in the vanguard of node.js development, was running into a nasty memory leak: a leak sufficiently acute that the service in question was having to be restarted on a regular basis, but not so much that it was reproducible in development. With the urgency high on their side, we again kicked around this seemingly hopeless problem, focussing on the concrete data that we had: core dumps exhibiting the high memory usage, obtained at our request via gcore(1) before service restart. Could we do anything with those?
As an aside, a few months ago, Joyent’s Dave Pacheco added some unbelievable postmortem debugging support for node. (If postmortem debugging is new to you, you might be interested in checking out my presentation on debugging production systems from QCon San Francisco. I had the privilege of demonstrating Dave’s work in that presentation — and if you listen very closely at 43:26, you can hear the audience gasp when I demonstrate the amazing ::jsprint.) But Dave hasn’t built the infrastructure for walking the data structures of the V8 heap from an MDB dmod — and it was clear that doing so would be brittle and error-prone.
As Dave, Brendan and I were kicking around increasingly desperate ideas (including running strings(1) on the dump — an idea not totally without merit — and some wild visualization ideas), a much simpler idea collectively occurred to us: given that we understood via Dave’s MDB V8 support how a given object is laid out, and given that an object needed quite a bit of self-consistency with referenced but otherwise orthogonal structures, what about just iterating over all of the anonymous memory in the core and looking for objects? That is, iterate over every single address, and see if that address could be interpreted as an object. On the one hand, it was terribly brute force — but given the level of consistency required of an object in V8, it seemed that it wouldn’t pick up too many false positives. The more we discussed it, the more plausible it became, but with Dave fully occupied (on another saucy project we have cooking at Joyent — more on that later), I roped up and headed into his MDB support for V8…
The result — ::findjsobjects — takes only a few minutes to run on large dumps, but provides some important new light on the problem. The output of ::findjsobjects consists of representative objects, the number of instances of that object and the number of properties on the object — followed by the constructor and first few properties of the objects. For example, here is the dump on a gcore of a (non-pathological) Joyent node-based facility:
> ::findjsobjects -v findjsobjects: elapsed time (seconds) => 20 findjsobjects: heap objects => 6079488 findjsobjects: JavaScript objects => 4097 findjsobjects: processed objects => 1734 findjsobjects: unique objects => 161 OBJECT #OBJECTS #PROPS CONSTRUCTOR: PROPS fc4671fd 1 1 Object: flags fe68f981 1 1 Object: showVersion fe8f64d9 1 1 Object: EventEmitter fe690429 1 1 Object: Credentials fc465fa1 1 1 Object: lib fc46300d 1 1 Object: free fc4efbb9 1 1 Object: close fc46c2f9 1 1 Object: push fc46bb21 1 1 Object: uncaughtException fe8ea871 1 1 Object: _idleTimeout fe8f3ed1 1 1 Object: _makeLong fc4e7c95 1 1 Object: types fc46bae9 1 1 Object: allowHalfOpen ... fc45e249 12 4 Object: type, min, max, default fc4f2889 12 4 Object: index, fields, methods, name fd2b8ded 13 4 Object: enumerable, writable, configurable, value fc0f68a5 14 1 SlowBuffer: length fe7bac79 18 3 Object: path, fn, keys fc0e9d21 20 5 Object: _onTimeout, _idleTimeout, _idlePrev, ... fc45facd 21 4 NativeModule: loaded, id, exports, filename fc45f571 23 8 Module: paths, loaded, id, parent, exports, ... fc4607f9 35 1 Object: constructor fc0f86c9 56 3 Buffer: length, offset, parent fc0fc92d 57 2 Arguments: length, callee fe696f59 91 3 Object: index, fields, name fc4f3785 91 4 Object: fields, name, methodIndex, classIndex fe697289 246 2 Object: domain, name fc0f87d9 308 1 Buffer:
Now, any one of those objects can be printed with ::jsprint. For example, let’s take fc45e249 from the above output:
> fc45e249::jsprint
{
type: number,
min: 10,
max: 1000,
default: 300,
}
Note that that’s only a representative object — there are (in the above case) 12 objects that have that same property signature. ::findjsobjects can get you all of them when you specify the address of the reference object:
> fc45e249::findjsobjects fc45e249 fc46fd31 fc467ae5 fc45ecb5 fc45ec99 fc45ec11 fc45ebb5 fc45eb41 fc45eb25 fc45e3d1 fc45e3b5 fc45e399
And because MDB is the debugger Unix was meant to have, that output can be piped to ::jsprint:
> fc45e249::findjsobjects | ::jsprint
{
type: number,
min: 10,
max: 1000,
default: 300,
}
{
type: number,
min: 0,
max: 5000,
default: 5000,
}
{
type: number,
min: 0,
max: 5000,
default: undefined,
}
...
Okay, fine — but where are these objects referenced? ::findjsobjects has an option for that:
> fc45e249::findjsobjects -r fc45e249 referred to by fc45e1e9.height
This tells us (or tries to) who is referencing that first (representative) object. Printing that out (with the “-a” option to show the addresses of the objects):
> fc45e1e9::jsprint -a
fc45e1e9: {
ymax: fe78e061: undefined,
hues: fe78e061: undefined,
height: fc45e249: {
type: fe78e361: number,
min: 14: 10,
max: 7d0: 1000,
default: 258: 300,
},
selected: fc45e3fd: {
type: fe7a2465: array,
default: fc45e439: [...],
},
...
So if we want to learn where all of these objects are referenced, we can again use a pipeline within MDB:
> fc45e249::findjsobjects | ::findjsobjects -r fc45e249 referred to by fc45e1e9.height fc46fd31 referred to by fc46b159.timeout fc467ae5 is not referred to by a known object. fc45ecb5 referred to by fc45eadd.ymax fc45ec99 is not referred to by a known object. fc45ec11 referred to by fc45eadd.nbuckets fc45ebb5 referred to by fc45eadd.height fc45eb41 referred to by fc45eadd.ymin fc45eb25 referred to by fc45eadd.width fc45e3d1 referred to by fc45e1e9.nbuckets fc45e3b5 referred to by fc45e1e9.ymin fc45e399 referred to by fc45e1e9.width
Of course, the proof of a debugger is in the debugging; would ::findjsobjects actually be of use on the Voxer dumps that served as its motivation? Here is the (elided) output from running it on a big Voxer dump:
> ::findjsobjects -v findjsobjects: elapsed time (seconds) => 292 findjsobjects: heap objects => 8624128 findjsobjects: JavaScript objects => 112501 findjsobjects: processed objects => 100424 findjsobjects: unique objects => 241 OBJECT #OBJECTS #PROPS CONSTRUCTOR: PROPS fe806139 1 1 Object: Queue fc424131 1 1 Object: Credentials fc424091 1 1 Object: version fc4e3281 1 1 Object: message fc404f6d 1 1 Object: uncaughtException ... fafcb229 1007 23 ClientRequest: outputEncodings, _headerSent, ... fafc5e75 1034 5 Timing: req_start, res_end, res_bytes, req_end, ... fafcbecd 1037 3 Object: aborted, data, end 8045475 1060 1 Object: fb0cee9d 1220 9 HTTPParser: socket, incoming, onHeadersComplete, ... fafc58d5 1271 25 Socket: _connectQueue, bytesRead, _httpMessage, ... fafc4335 1311 16 ServerResponse: outputEncodings, statusCode, ... fafc4245 1673 1 Object: slab fafc44d5 1702 5 Object: search, query, path, href, pathname fafc440d 1784 14 Client: buffered_writes, name, timing, ... fafc41c5 1796 3 Object: error, timeout, close fafc4469 1811 3 Object: address, family, port fafc42a1 2197 2 Object: connection, host fbe10625 2389 2 Arguments: callee, length fafc4251 2759 15 IncomingMessage: statusCode, httpVersionMajor, ... fafc42ad 3652 0 Object: fafc6785 11746 1 Object: oncomplete fb7abc29 15155 1 Object: buffer fb7a6081 15155 3 Object: , oncomplete, cb fb121439 15378 3 Buffer: offset, parent, length
This immediately confirmed a hunch that Matt had had that this was a buffer leak. And for Isaac — who had been working this issue from the Gedanken side and was already zeroing in on certain subsystems — this data was surprising in as much as it was so confirming: he was already on the right path. In short order, he nailed it, and the fix is in node 0.6.17.
The fix was low risk, so Voxer redeployed with it immediately — and for the first time in quite some time, memory utilization was flat. This was a huge win — and was the reason for Matt’s tantalizing tweet. The advantages of this approach are that it requires absolutely no modification to one’s node.js programs — no special flags and no different options. And it operates purely postmortem. Thanks to help from gcore(1), core dumps can be taken over time for a single process, and those dumps can then be analyzed off-line.
Even with ::findjsobjects, debugging node.js memory leaks is still tough. And there are certainly lots of improvements to be made here — there are currently some objects that we do not know how to correctly interpret, and we know that we know that we can improve our algorithm for finding object references — but this shines a bright new light into what had previously been a black hole!
If you want to play around with this, you’ll need SmartOS or your favorite illumos distro (which, it must be said, you can get simply by provisioning on the Joyent cloud). You’ll need an updated v8.so — which you can either build yourself from illumos-joyent or you can download a binary. From there, follow Dave’s instructions. Don’t hesitate to ping me if you get stuck or have requests for enhancement — and here’s to flat memory usage on your production node.js service!
Alerts are great for not only letting you know when there's some kind of hardware event, but they can also be pro-active and let you know there's a bottleneck coming BEFORE it happens. Check these out. There are two kinds of Alerts in the ZFSSA. When you go to Configuration-->Alerts, you fist see the plus sign by the "Alert Actions" section. These are pretty self-explanatory and not what I'm talking about today. Click on the "Threshold Alerts", and then click the plus sign by those.
This is what I'm talking about. The default one that comes up, "CPU: Percent Utilization" is a good one to start with. I don't mind if my CPUs go to 100% utilized for a short time. After all, we bought them to be used, right? If they go over 90% for over 10 minutes, however, something is up, and maybe we have workloads on this machine it was not designed for, or we don't have enough CPUs in the system and need more. So we can setup an alert that will keep an eye on this for us and send us an email if this were to occur. Now I don't have to keep watching it all the time. For an even better example, keep reading...
What if you want to keep your eyes on whether your Readzillas or Logzillas are being over-utilized? In other words, do you have enough of them? Perhaps you only have 2 Logzillas, and you think you may be better off with 4, but how do you prove it? No problem. Here in Threshold Alerts, click on the Threshold drop-down box, and choose your "Disk: Percent Utilization for Disk: Jxxxxx 013" choice, which is my Logzilla drive in the Jxxxxx tray.
Wait. What's that? You don't have a choice in your drop-down for the Threshold item you are looking for, such as an individual disk?
Well, we will have to fix that.
Leave Alerts for now, and join me over in Analytics. Start with a worksheet with "Disk: Percent utilization broken down by Disk" chart. You do have this, as it's already one of your built-in datasets.
Now, expand it so you can see all of your disks, and find one of your Readzilla or Logzilla drives. (Hint: It will NOT be disk 13 like my example here. Logzillas are always in the 20, 21, 22, or 23 slots of a disk tray. Go to your Configuration-->Hardware screens and you can easily find out which drives are which for your system).
Now, click on that drive to highlight it, like this:
Click on the Drill Button, and choose to drill down on that drive as a raw statistic. You will now have a whole new data chart, just for that one drive.
Don't go away yet. You now need to save that chart as a new dataset, which will keep it in your ZFSSA analytic metrics forever. Well, until you delete it.
Click on the "Save" button, the second to last button on that chart. It looks like a circle with white dots on it (it's supposed to look like a reel-to-reel tape spindle).
Now go to your "Analytics-->Datasets", and you will see a new dataset in there for it.
Go back to your Threshold Alerts, and you will now be able to make an alert that will tell you if this specific drive goes over 90% for more than 10 minutes. If this happens a lot, you probably need more Readzillas or Logzillas.
I hope you like these Alerts. They may take some time to setup at first, but in the long run you may thank yourself. It might not be a bad idea to send the email alerts to a mail distribution list, instead of a single person who may be on vacation when the alert is hit. Enjoy.
In this episode of the "Oracle Solaris: In a Class By Itself" podcast series, the focus is a bit more technical. I was interviewed by host Charlie Boyle, Senior Director of Solaris Product Marketing. We talked about a new feature in Oracle Solaris 11: immutable zones. Those are read-only root zones for highly secure deployment scenarios.
See also my previous blog post on Enctypted Immutable Zones.
Join the Oracle hardware and software team in booth 1323 at the DISA Partnership Conference, May 7-10 in Tampa, FL. A wide variety of Oracle technology and staff will be available to answer your questions and offer solutions to your information processing problems.
Oracle's President Mark Hurd will deliver a keynote address.
On display will be:
Come see us across from the DISA pavilion.
Hi, this is Eric Maurice.
Oracle just released Security Alert CVE-2012-1675 to address the “TNS Listener Poison Attack” in the Oracle Database. With a CVSS Base Score of 7.5, this vulnerability is remotely exploitable without authentication, and if successfully exploited, can result in a full compromise of the targeted Database.
In the April 2012 Critical Patch Update, Oracle provided Security-in-Depth recognition to Joxean Koret. As stated in the Critical Patch Update advisories, “People are recognized for Security-In-Depth contributions if they provide information, observations or suggestions pertaining to security vulnerability issues that result in significant modification of Oracle code or documentation in future releases, but are not of such a critical nature that they are distributed in Critical Patch Updates.”
As stated in previous blog entries, Oracle fixes vulnerability first in the main code line, and then tries to backport fixes through the Critical Patch Update program for exploitable vulnerabilities that were externally reported. In certain instances, such backporting is very difficult or impossible because of the amount of code change required, or because the fix would create significant regressions, or because there is no reasonable way to automate the application of the fix (for example when user interaction is required to change configuration parameters).
Shortly after the release of the Critical Patch Update, mistakenly assuming that the issue had been backported through the CPU, Joxean Koret, the initial reporter of this vulnerability, fully disclosed its details, initially stating that it had been fixed by Oracle, then after realizing that it had not been fixed in current releases, reported the vulnerability as a “0-day.”
As a result of this disclosure, Oracle has issued Security Alert CVE-2012-1675 to provide customers with a number of technical measures to provide effective defense against this vulnerability in all deployment scenarios.
Customers on single-node configurations (i.e., non Real Application Cluster (RAC) customers) should refer to the My Oracle Support Note titled “Using Class of Secure Transport (COST) to Restrict Instance Registration” (Doc ID 1453883.1) to limit registration to the local node and the IPC protocol through the COST (Class Of Secure Transport) feature in the listener.
RAC and Exadata customers should refer to the My Oracle Support Note “Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle RAC” (Doc ID 1340831.1) to implement similar COST restrictions.
Note that implementing COST restrictions in RAC environments require the use of SSL/TLS encryption. Such network encryption features were previously only available to customers who were licensed for Oracle Advanced Security. However, RAC customers who were previously not licensed for Oracle Advanced Security need not be concerned about a licensing restriction as Oracle has updated its licensing to allow these customers the use of these features (namely SSL and TLS) to protect themselves against vulnerability CVE-2012-1675. In other words, Oracle has added Oracle Advanced Security SSL/TLS to the Enterprise Edition Real Application Clusters (Oracle RAC) and RAC One Node options, and added Oracle Advanced Security SSL/TLS to the Oracle Database Standard Edition license when used with the Real Application Clusters.
Considering that the technical details of vulnerability CVE-2012-1675 have now widely been distributed, Oracle highly recommends that customers make the configuration changes documented in the above mentioned My Oracle Support Notes as soon as possible. Customers should also feel free to contact Oracle Support if they have questions or concerns.
For More Information:
The universe is change; our life is what our thoughts make it.
(Marcus Aurelius)
If you have a job in IT (and who among my readers hasn't?), then it is going to fundamentally change soon.
Why?
In my own job, I see the full spectrum from where IT innovation is created to the very last laggards who are still depending a lot on mainframes and other ancient technology. Some things in IT are new (like, every week there's a new startup/technology/trend that is shaking up the industry), and some things are just repetitions of stuff that has happened before, albeit in slightly different colors.
So now, the world of IT as we know it is changing (again) and this time, change will impact organizations, roles and jobs.
Let's dive a little bit into what's happening. Don't worry, change is good, but only if you prepare for it.
There are three trends that are hitting the IT landscape right now. They are more or less independent, but they also complement each other:
Information technology is just a stack of technologies. In its simplest form, we see hardware at the bottom, and software at the top. Over time, the stack has diversified into a more complex building where IT is composed of:
For each of these components, every IT department needs to decide whether to build something from scratch, or just buy a pre-engineered solution.
A long time ago, IT shops built their own storage and server components, or they built their own databases and middleware, but that was very quickly delegated to IT vendors. Today, IT shops build systems out of components.
But that is changing:
So if your job/role/task has been to evaluate host bus adapters and their compatibilities with individual servers and SAN components, chasing down storage firmware patches so your server's OS doesn't hiccup too badly when confronted with a certain type of disk or if you run a test lab that regularly evaluates storage, server or networking components, then it's time to rethink what you're doing and whether this is going to continue being a good use of your time.
The trend here is obvious: Smaller parts are more and more integrated at the factory level, bigger parts are delivered to IT organizations, the point of deployment and integration is rising up the stack over time.
Today's part granularity is the rack, not the component. And if racks are too much of a hassle for you, you can get rid of all this hardware stuff altogether by using cloud computing.
Ultimately, the whole purpose of running IT is to provide business services: A credit card billing service or maybe a blogging service or even an online dog food store. Every such service is composed of components across the whole IT stack.
Unfortunately, this creates a lot of dependencies: If you want to upgrade your storage, you need to plan for it, otherwise your credit card service would suffer, if your server needs a patch, you need to make sure the blogging software running on top of this server is moved elsewhere first, if you're running a consolidation project, you better schedule a few meetings with your dog food store department to make sure the transition is accurately planned, managed and executed without disrupting your service and so on.
But: The more stuff is dependent on more other stuff, the stiffer your organization is, the longer stuff takes time to complete and the more errors can happen along the way. Agility suffers.
The solution is disintermediation:
And disintermediation is hitting again, this time thanks to Engineered Systems and cloud computing. Whether you're operating a private cloud (based on Engineered Systems or not) or you're using a public cloud, your IT operations are now more and more independent from your IT services:
While IT ops is focusing more and more on running a private cloud infrastructure that is application agnostic (or maybe their company has decided to entirely offload IT into the cloud), the remaining pieces of IT that are closer to developing and managing IT at the application/service level enjoy more independence, more control but also more responsibility for their services.
This trend has been known as “DevOps” for quite a while (and I highly recommend Ben Rockwood's blog posts on DevOps, or this piece by Adrian Cockcroft of Netflix). Werner Vogels, Amazon CTO put it into his now famous words: “You build it, you run it!”.
What does this mean for your job? Well, if you're working in IT and you're now evolving into a more cloud based model, you need to decide whether you want to be part of the cloud operations bit (which is going to be much more automated and hence will offer less room for creativity), or maybe it is time to huddle up closer with your developers and service operators: They now have a need for some operating system and HA and network savvy guys that help them do this “you run it” thing, now that the cloud is here.
With all this change (from assembling components by hand to automated data centers, from silo IT to cloud computing and so on), there comes a lot of new opportunity: New stuff needs to be figured out which requires creativity, new tools and technologies emerge which need to be explored, new roles and responsibilities are created almost daily.
Here's some food for thought:
As new IT concepts like virtualization, Engineered Systems, Cloud Computing, DevOps, new services, patterns and languages emerge, they force IT organizations to re-think and adapt roles, responsibilities and jobs to the new reality. Change is a constant in IT, and the current times are likely to see a lot more change than we have seen before.
Try to climb up the IT stack towards places that are closer to where your company creates value, and where your creativity makes a difference, as commoditization, automation and outsourcing in the industry climbs up behind you and is now hitting the system integration level.
Re-think how your company's value creation processes map to business processes and ultimately IT processes and how new ways of disintermediation affect your processes and your architecture methodology. If DevOps is the new ITIL, what would a new version of TOGAF look like?
Finally, try to learn something new from the rapidly changing web culture, be it a new way to look at work, a new scripting language or a new API.
Nobody knows how the future is going to look like, but the more you dig into new stuff, the closer you'll be to whatever the winners of the future are.
Here are a few books that may be interesting, helpful or just fun to read as you re-invent your job (and yes, these are affiliate links. They help me pay for hosting costs and they'll cost you nothing, we both win!):
(If you don't see anything, then you have an ad blocker enabled. Trust me, at least this time, it'll be ok to switch it off :).)
What changes do you see in your organization as your IT department evolves?
How do you prepare for change?
What other tips, resources and books do you recommend to stay on top of changes in IT?
Share your insights in the comments!
Street sign photo by Flickr-user CBS_Fan, used and modified under CC license.
DTrace first peered into Java in early 2005 thanks to an early prototype by Jarod Jenson that led eventually to the inclusion of USDT probes in the HotSpot JVM. If you want to see where, say, the java.net.SocketOutputStream.write() method is called, you can simply run this DTrace script:
hotspot$target:::method-entry
/copyinstr(arg1, arg2) == "java/net/SocketOutputStream" &&
copyinstr(arg3, arg4) == "write"/
{
jstack(50, 8000);
}
And that will work as long as you rememember to start your JVM with the -XX:+ExtendedDTraceProbes option or you use the jinfo utility to enable it after the fact. And as long as you don’t mind a crippling performance penalty (hint: you probably do).
Inspired by dtrace.conf a few weeks ago, I wanted to sketch out what the real Java provider would look like:
java$target:java.net.SocketOutputStream:write:entry
{
jstack(50,8000);
}
And check it out:
# jdtrace.pl -p $(pgrep java) -n 'java$target:java.net.SocketOutputStream::entry{ jstack(50,8000); }'
dtrace: script '/tmp/jdtrace.19092/jdtrace.d' matched 0 probes
CPU ID FUNCTION:NAME
0 64991 Java_com_sun_btrace_BTraceRuntime_dtraceProbe0:event
libbtrace.so`Java_com_sun_btrace_BTraceRuntime_dtraceProbe0+0xbb
com/sun/btrace/BTraceRuntime.dtraceProbe0(Ljava/lang/String;Ljava/lang/String;II)I
com/sun/btrace/BTraceRuntime.dtraceProbe(Ljava/lang/String;Ljava/lang/String;II)I
com/sun/btrace/BTraceUtils$D.probe(Ljava/lang/String;Ljava/lang/String;II)I
com/sun/btrace/BTraceUtils$D.probe(Ljava/lang/String;Ljava/lang/String;)I
java/net/SocketOutputStream.$btrace$jdtrace$probe1(Ljava/lang/String;Ljava/lang/String;)V
java/net/SocketOutputStream.write([BII)V
sun/nio/cs/StreamEncoder.writeBytes()V
sun/nio/cs/StreamEncoder.implFlushBuffer()V
sun/nio/cs/StreamEncoder.implFlush()V
sun/nio/cs/StreamEncoder.flush()V
java/io/OutputStreamWriter.flush()V
java/io/BufferedWriter.flush()V
java/io/PrintWriter.newLine()V
java/io/PrintWriter.println()V
java/io/PrintWriter.println(Ljava/lang/String;)V
com/delphix/appliance/server/ham/impl/HAMonitorServerThread.run()V
java/util/concurrent/ThreadPoolExecutor$Worker.runTask(Ljava/lang/Runnable;)V
java/util/concurrent/ThreadPoolExecutor$Worker.run()V
java/lang/Thread.run()V
StubRoutines (1)
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x21d
libjvm.so`__1cCosUos_exception_wrapper6FpFpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v2468_v_+0x27
libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_+0x149
libjvm.so`__1cMthread_entry6FpnKJavaThread_pnGThread__v_+0x113
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2c6
libjvm.so`java_start+0x1f2
libc.so.1`_thrp_setup+0x9b
libc.so.1`_lwp_start
Obviously there's something fishy going on. First, we're using perl -- the shibboleth of fake-o-ware -- and there's this BTrace stuff in the output.
BTrace is a dynamic instrumentation tool for Java; it is both inspired by DTrace and contains some DTrace integration. The perl script above takes the DTrace syntax and generates a DTrace script and a BTrace-enabled Java source file.
Like DTrace, BTrace lets you specify the points of instrumentation in your Java program as well as the actions to take. Here's what our generated source file looks like.
import com.sun.btrace.annotations.*;
import static com.sun.btrace.BTraceUtils.*;
@BTrace
public class jdtrace {
@OnMethod(clazz="java.net.SocketOutputStream", method="write", location=@Location(Kind.ENTRY))
public static void probe1(@ProbeClassName String c,
@ProbeMethodName String m) {
String name = "entry";
String p = Strings.strcat(c, Strings.strcat(":",
Strings.strcat(m, Strings.strcat(":", name))));
D.probe(p, "");
}
}
Note that we specify where to trace (this can be a regular expression), and then take the action of joining the class, method, and "entry" string into a single string that we pass to the D.probe() method that causes a BTrace USDT probe to fire.
Here's what the D script looks like:
btrace$target:::event
{
this->__jd_arg = copyinstr(arg0);
this->__jd_mod = strtok(this->__jd_arg, ":");
this->__jd_func = strtok(NULL, ":");
this->__jd_name = strtok(NULL, ":");
}
btrace$target:::event
/((this->__jd_mod == "java.net.SocketOutputStream" &&
this->__jd_func == "write" &&
this->__jd_name == "entry"))/
{
jstack(50,8000);
}
It's pretty simple. We parse the string that was passed to D.probe(), and disassemble it into the DTrace notion of module, function, and name. We then use that information so that the specified actions are executed as appropriate (we could have specified different Java methods to probe, and different actions to take for each). Here's the code if you're interested.
This isn't the real Java provider, but is it close enough? Unfortunately not. The most glaring problem is that BTrace sometimes renders my Java process unresponsive. Other times it leaves instrumentation behind with no way of extracting it. The word "safe" appears as the third word on the BTrace website ("BTrace is safe"), but apparently there's still some way to go to achieve the requisite level of safety.
BTrace is an interesting tool for examining Java programs, but one obvious obstacle is that the programs are pretty cumbersome to write. With BTrace, we should be able to write a simple one-liner to see where we are when the java.net.SocketOutputStream.write() method is called, but instead we have to write a fairly cumbersome program:
import com.sun.btrace.annotations.*;
import static com.sun.btrace.BTraceUtils.*;
@BTrace
public class TraceWrite {
@OnMethod(clazz="java.net.SocketOutputStream", method="write", location=@Location(Kind.ENTRY))
public static void onWrite() {
jstack();
}
}
DTrace-inspired syntax would let users iterate much more quickly:
$ dbtrace -p $(pgrep -n java) -n 'java.net.SocketOutputStream:write:entry{ jstack(); }'
java.net.SocketOutputStream.write(SocketOutputStream.java)
sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276)
sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
java.io.BufferedWriter.flush(BufferedWriter.java:236)
java.io.PrintWriter.newLine(PrintWriter.java:438)
java.io.PrintWriter.println(PrintWriter.java:585)
java.io.PrintWriter.println(PrintWriter.java:696)
com.delphix.appliance.server.ham.impl.HAMonitorServerThread.run(HAMonitorServerThread.java:56)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
With BTrace, you can trace nearly arbitrary information about a program's state, but instead of doing something like this:
dbtrace -p $(pgrep -n java) -n 'java.net.SocketOutputStream:write:entry{ printFields(this.impl); }'
You have to do this:
import com.sun.btrace.annotations.*;
import com.sun.btrace.AnyType;
import static com.sun.btrace.BTraceUtils.Reflective.*;
@BTrace
public class TraceWrite {
@OnMethod(clazz="java.net.SocketOutputStream", method="write", location=@Location(Kind.ENTRY))
public static void onWrite(@Self Object self) {
Object impl = get(field(classOf(self), "impl"), self);
printFields(impl);
}
}
$ ./bin/btrace $(pgrep -n java) TraceWrite.java
{server=null, port=1080, external_address=null, useV4=false, cmdsock=null, cmdIn=null, cmdOut=null, applicationSetProxy=false, timeout=0, trafficClass=0, shut_rd=false, shut_wr=false, socketInputStream=java.net.SocketInputStream@9993a1, fdUseCount=0, fdLock=java.lang.Object@ab5443, closePending=false, CONNECTION_NOT_RESET=0, CONNECTION_RESET_PENDING=1, CONNECTION_RESET=2, resetState=0, resetLock=java.lang.Object@292936, fd1=null, anyLocalBoundAddr=null, lastfd=-1, stream=false, socket=Socket[addr=/127.0.0.1,port=38832,localport=8765], serverSocket=null, fd=java.io.FileDescriptor@50abcc, address=/127.0.0.1, port=38832, localport=8765, }
BTrace needs a language that enables rapid iteration — piggybacking on Java is holding it back — and it needs some hard safety guarantees. With those, many developers and support engineers would use BTrace as part of their daily work — we certainly would here at Delphix.
Back to DTrace. Even with a useable solution for Java only, the ability to have lightweight and focused tracing for Java (and other dynamic languages) could be highly valuable. We’ll see how far BTrace can take us.
Let's talk about your Routing Table.
I have never installed a ZFSSA, ever, without having to edit this table. If you believe that you do not need to edit your routing table, then you are wrong.
:) Ok, maybe not. Maybe you only have your ZFSSA connected to one network with only a few systems on it. I guess it's possible. Even in my simulator, however, I had to edit the routing table so I could use it no matter how I had my laptop connected, at home over a VPN or at work or using a public Wifi. So I'm going to bet a nice dinner that you, or someone, should be checking this out.
First things first. I'm going to assume you have a cluster. I try really hard to only sell clusters, but yes I know there are plenty of single-nodes out there too. Single-node people can skip these first two paragraphs. It's very important in your cluster to have a 1GigE management interface to each of the two controllers. You really want to be able to manage each controller, even when one of them is down, right? So best practice is to use the 'igb0' port for controller 1 management and to use the 'igb1' port for controller 2 management. It's important to make these ports 'Private' in the cluster configuration screen, so they do NOT failover to the other controller when a cluster takeover takes place for whatever reason. Igb0 and igb1 are two of the four built-in 1GigE ports. You can still use igb2 and igb3 for data, either alone or as an aggregate, and don't make them private, so they DO failover in a cluster takeover event. Now go to your remote workstation, which may be over a different subnet, and you should be able to ping and connect to Controller 1 using igb0.
Now, back to the routing table. You have probably noticed that you can not ping or connect to the other controller, and you think something is wrong. Not to worry, everything is fine. You just need to tell your routing table, which is shared between the heads, how to talk to that other port, igb1. You see, you have a default route setup already for port igb0, that's why it works. Your new, private, igb1 however, does not know how to speak back to your remote system you are now using to manage via the BUI from a different subnet. So, make a new default route for igb1 and point it to the default gateway, which is the router it needs to use in order to cross subnets. See the picture below. Note how I have a default route for "ZFS1-MGMT" for port igb0. This shows a green light because I'm currently on ZFS1, and it sees this port just fine. I also have a default route for "ZFS2-MGMT" from port igb1. This route has a blue light, showing it as inactive. That's because this controller, ZFS1, has nothing plugged into it's igb1 port. That's perfect. Hit "Apply". Now count to 10. Now from your remote host, go ahead and ping or connect to Controller 2, and it works!!! This is because your controllers share a routing table, and when you added that igb1 route, it propagated over to the other controller, where igb1 is plugged in, and that route has a green light over there and it works fine. You will see from Controller 2's point of view that igb1 has a green light and igb0 has a blue light. (continued below the picture)
Now it's time to setup any static routes you may need. If you have different subnets for your 1GigE management and your IB or 10GigE data (a very good idea), then you will need to make these. It's important to have routes for this, as you do not want data coming in over the 10GigE pipe, but then returning over the 1GigE pipe, right? That will happen if this is not setup correctly. Make your routes, as the picture example shows with a 10Gig aggragate here we called "Front-end-IP". Any traffic coming in from subnet 172.20.69 will use this pipe.
Lastly, check your multi-homing model button up top. I like 'Adaptive'. Loose is the default, and makes it so your packets can traverse your routes, even though they may go over the wrong route, so it seems like your system is working. This can very well be an illusion. Your ping may work, but it may be coming from the wrong interface, as "Loose" basically means the ZFSSA just doesn't care or enforce any rules. "Strict", on the other hand, is great if you want total enforcement. If you are very good with your routes, and are positive you have it right, and want to ensure that a packet never goes the wrong way, even if that means dropping the packet, then use Strict. I'm using Adaptive here, which is a happy medium. From the help file: The "Adaptive" choice will prefer routes with a gateway address on the same subnet as the packet's source IP address: 1) An IP packet will be accepted on an IP interface so long as its destination IP address is up on the appliance. 2) An IP packet will be transmitted over the IP interface tied to the route that most specifically matches an IP packet's destination address. If multiple routes are equally specific, prefer routes that have a gateway address on the same subnet as the packet's source address. If no eligible routes exist, drop the packet.
Update 4/23/12- My colleague, Darius (https://blogs.oracle.com/si/), rightfully wanted me to point out how important it was to setup a static route for replication. You do not want replication to go over a private management port by mistake, as this will cause it to fail when one controller or the other goes down for maintenance.
I hope this helps. Routing can be fun.
If you are computing both the sine and cosine of an angle, then you will be twice as quick if you call sincos() than if you call cos() and sin() independently:
#include
int main()
{
double a,b,c;
a=1.0;
for (int i=0;i<100000000;i++) { b=sin(a); c=cos(a); }
}
$ cc -O sc.c -lm
$ timex ./a.out
real 19.13
vs
#include
int main()
{
double a,b,c;
a=1.0;
for (int i=0;i<100000000;i++) { sincos(a,&b,&c); }
}
$ cc -O sc.c -lm
$ timex ./a.out
real 9.80
I've talked about building 64-bit libraries with position independent code. When building 64-bit applications there are two options for the code that the compiler generates: -xcode=abs64 or -xcode=abs44, the default is -xcode=abs44. These are documented in the user guides. The abs44 and abs64 options produce 64-bit applications that constrain the code + data + BSS to either 44 bit or 64 bits of address.
These options constrain the addresses statically encoded in the application to either 44 or 64 bits. It does not restrict the address range for pointers (dynamically allocated memory) - they remain 64-bits. The restriction is in locating the address of a routine or a variable within the executable.
This is easier to understand from the perspective of an example. Suppose we have a variable "data" that we want to return the address of. Here's the code to do such a thing:
extern int data;
int * address()
{
return &data
}
If we compile this as a 32-bit app we get the following disassembly:
/* 000000 4 */ sethi %hi(data),%o5 /* 0x0004 */ retl ! Result = %o0 /* 0x0008 */ add %o5,%lo(data),%o0
So it takes two instructions to generate the address of the variable "data". At link time the linker will go through the code, locate references to the variable "data" and replace them with the actual address of the variable, so these two instructions will get modified. If we compile this as a 64-bit code with full 64-bit address generation (-xcode=abs64) we get the following:
/* 000000 4 */ sethi %hh(data),%o5 /* 0x0004 */ sethi %lm(data),%o2 /* 0x0008 */ or %o5,%hm(data),%o4 /* 0x000c */ sllx %o4,32,%o3 /* 0x0010 */ or %o3,%o2,%o1 /* 0x0014 */ retl ! Result = %o0 /* 0x0018 */ add %o1,%lo(data),%o0
So to do the same thing for a 64-bit application with full 64-bit address generation takes 6 instructions. Now, most hardware cannot address the full 64-bits, hardware typically can address somewhere around 40+ bits of address (example). So being able to generate a full 64-bit address is currently unnecessary. This is where abs44 comes in. A 44 bit address can be generated in four instructions, so slightly cuts the instruction count without practically compromising the range of memory that an application can address:
/* 000000 4 */ sethi %h44(data),%o5 /* 0x0004 */ or %o5,%m44(data),%o4 /* 0x0008 */ sllx %o4,12,%o3 /* 0x000c */ retl ! Result = %o0 /* 0x0010 */ add %o3,%l44(data),%o0
A couple of folks have posted comments on old blog entries complaining that they can't resolve links to old SunSolve content.
SunSolve was decommissioned as part of the integration into Oracle. Documentation which is still relevant was migrated. The URIs changed in the process. So did the document IDs.
On the basis that it's better to teach a man how to fish than to just give him a fish, here's a tip on how to do find the current links to such migrated documentation. At least, this is how I do it:
If you're looking to find the current URI for a document for which you have a stale URI such as http://sunsolve.sun.com/search/document.do?assetkey=1-79-250526.1-1 , simply search MOS using the "core" document number in the old SunSolve URI - for example "250526" in this example. Enter this in the "Search Knowledge Base" search box.
If the document is still relevant, MOS will usually return the corresponding new Document ID. For example, in this case Document 1019995.1. The top line of the document 1019995.1 states "Migrated ID: 250526".
I've updated the Doc IDs for a number of my old blog postings. Apologies for the inconvenience caused.
Hi, this is Eric Maurice.
Oracle has just released the April 2012 Critical Patch Update. This Critical Patch Update provides 88 new security fixes across the following product families: Oracle Database Server, Oracle Fusion Middleware, Oracle Enterprise Manager Grid Control, Oracle E-Business Suite, Oracle Supply Chain Products Suite, Oracle PeopleSoft Enterprise, Oracle FLEXCUBE, Oracle Siebel Clinical Trial Management System, Oracle Primavera, Oracle Sun products suite, and Oracle MySQL.
Of the 88 new vulnerabilities, 6 directly affect Oracle Database Server. The highest CVSS Base Score for these Database Server vulnerabilities is 9.0. This Base Score affects the Oracle Spatial component on Windows platforms (on non-Windows platforms, i.e., Linux, Unix, the CVSS Base Score is 6.5). In addition, 6 Enterprise Manager Grid Control fixes may be relevant to Database Server deployments. The highest CVSS Base Score for the Enterprise Manager Grid Control vulnerabilities is 5.8; but 4 of the 6 vulnerabilities can be remotely exploitable without authentication. Therefore, Oracle highly recommends that these fixes be applied as soon as possible.
This Critical patch Update also includes 11 new security fixes for Oracle Fusion Middleware. The highest CVSS Base Score for these Fusion Middleware vulnerabilities is 10.0 (for vulnerability CVE-2012-1695). This score affects a series of vulnerabilities in the Java Runtime Environment that are applicable to JRockit. Starting again with this Critical Patch Update, JRockit fixes will no longer be provided with the Critical Patch Update for Java SE, but be provided in “the normal” Critical Patch Update along with other Oracle Fusion Middleware fixes.
This Critical Patch Update provides the following application security fixes: 4 for Oracle E-Business Suite, 5 for Oracle Supply Chain Products Suite, 15 for Oracle PeopleSoft Enterprise, 2 for Siebel Clinical Trial Management System, 17 for Oracle FLEXCUBE, and 1 for Oracle Primavera Enterprise Project Management.
Finally, this Critical Patch Update provides 15 new security fixes for the Oracle Sun Products Suite (including Oracle Grid Engine, Oracle Glassfish Enterprise Server, Oracle Solaris, etc.) and 6 new security fixes for Oracle MySQL.
While a great amount of caution is required when analyzing the content of the Critical Patch Updates in an attempt to identify potential trends; I believe the content of this Critical Patch Update is consistent with the views expressed in previous blog entries: Oracle Software Security Assurance activities tend to result in lowering the number of exploitable security bugs in most mature product lines (that is the product lines who have implemented Oracle secure development practices for the longest time), and as a result we see a downward trend in the number of fixes for these product lines. On the other hand, newly acquired product lines often experience relatively large number of security fixes in the Critical Patch Updates. This is due in part to the increased visibility these products may get as a result of their acquisition by Oracle, as well as development’s access to an extended toolset (e.g., security scanning tools) and increased executive attention around security matters as a result of joining Oracle.
For More Information:
The April 2012 Critical Patch Update Advisory is located at http://www.oracle.com/technetwork/topics/security/cpuapr2012-366314.html
More information about Oracle Software Security Assurance is located at http://www.oracle.com/us/support/assurance/index.html
| Agenda | |
| 9:00 a.m. PDT | Keynote: Oracle Solaris - Strategy and Update Markus Flierl, Vice President, Oracle Solaris Engineering |
| 9:40 a.m. PDT | Oracle Solaris 11: Extreme Engineering - A Technical Update Dan Price, Senior Principal Product Engineer, Oracle Solaris Engineering Bart Smaalders, Senior Principal Product Engineer, Oracle Solaris Engineering |
| 10:20 a.m. PDT | Customers and Partners: Why We Moved to Oracle Solaris 11 A discussion of the reasons why businesses and commercial software developers have adopted Oracle Solaris 11, from the people responsible for these decisions |
| 11:00 a.m. PDT | Oracle Solaris: Core to the Oracle Systems Strategy John Fowler, Executive Vice President of Systems, Oracle |
This is pretty sweet. The new SPC2 benchmark came out last week, and the 7420 not only came in 2nd of ALL speed scores, but came in #1 for price per MBPS.
Check out this table. The 7420 score of 10,704 makes it really fast, but that's not the best part. The price one would have to pay in order to beat it is ridiculous. You can go see for yourself at http://www.storageperformance.org/results/benchmark_results_spc2
The only system on the whole page that beats it was over twice the price per MBPS. Very sweet for Oracle.
So let's see, the 7420 is the fastest per $.
The 7420 is the cheapest per MBPS.
The 7420 has incredible, built-in features, management services, analytics, and protocols. It's extremely stable and as a cluster has no single point of failure. It won the Storage Magazine award for best NAS system this year.
I see all of these issues crop up regularly. Number 1 just takes time and education. Number 3 takes time with our new, better, and growing support team. many of them came from Oracle and there were growing pains when they went from a straight software-model to having to also support hardware. Number 2 is tricky, but it's the job of the sales teams to break through the internal politics and help their clients see the value in oracle hardware systems. Benchmarks like this will help.
I’m highly introspective… far more than I would actually like to be. I’m one of those strange individuals to whom if you said “Do you realize your being a jerk right now?” I’d actually admit “Yes, I’m sorry about that, I’m trying to find a way to rectify it unsuccessfully.”
Despite that obsessive level of awareness, nothing can tell you more about who you are then your children. In particular, by observing things your children do that you never taught them, they just started doing of their own accord because “it seemed right”. Genetics at work.
I fight frequently with people about documenting processes. But maybe I’m just anal? Then the other day my son comes to me and shows me this:
This is Glenn, my eldest son (6 years old). He wanted some lemonade, but mom and I were busy. He decided it might help if he simplified his request into a process. You can see here that we start with a bottle of lemonade, then we pour it into a glass, then WHAMO! we have our amazingly refreshing beverage to enjoy. It is the perfect process with an input, output, and processing in the middle. Brilliant, and he hasn’t even been to business school yet. How much simpler does process get?
What about policy? Policy is just a business word for “rules”, nothing more. In my opinion, the worlds most amazing and effective policy is this one:
That yellow line is policy. Its not a brick wall, but we treat it like one. Thanks to that little bit of paint two cars can drive towards each other at 70 MPH, passing with only 6 ft between them, without fear. It doesn’t get simpler or more powerful than that.
Parents and authority figures in general, tend to layer into a child the concept of right and wrong as absolutes. Take the cookie and you shall be punished, so don’t take the cookie! All throughout our culture we do this, define a rules and corresponding punishments. The result is a general fear of rules, because they are seemingly there for the sole purpose of justifying punishment.
Any rule, any law, any policy, can be viewed as a guide or as a guillotine. When I asked many of my peers what they thought about policy a surprising number quickly answered “Its there so that you can fire people.” Its shocking how many people believe that. One would think that policy is there to enforce lessons learned in the past, as a guide for decision making, pre-computed solutions to problems which might be difficult to conflicting. So then why is it that they are considered simply a justification for punishment? Inconsistency of course… everyone seems to ignore, discount, or outright disregard policy on a day-to-day basis and it only comes to peoples attention when someone is being called out.
Policy and process are wonderful things. At least, they can be. They are the means by which we share knowledge within an organization. Common tasks, problems, and dilemmas can be quickly handled in a tried and true way, consistent throughout the organization, because we have policy and process. But in order for them to work, there are some ground rules, if you don’t follow them they are doomed to be the millstones of frustration most of us see them as:
The last point is the hardest. Knowledge management is still something we’re shitty at. Wiki’s have helped a lot over the last decade by making everything searchable and empowering everyone to update documents quickly and easily. But the fundamental problem is that of scaling. Not scaling the infrastructure but of the human mind. Many a sci-fi story has depicted the person who desire to know everything, and when the wish was granted, their head promptly exploded in one way or another. In many large companies when you hire on you’ll receive a book or binder with all the company policies… did you read it? Of course not: tl:dr.
Thus, what we’re really talking about here is culture. Genetics. Your children get them from you in the blood, but in a company we must teach them to others through words and actions. Preferably when employees are new, through on the job training/mentoring/tasking. Will you ignore policy and process? If you don’t care, they are likely useless crap anyways, and everyone can fend for themselves and hopefully get it right. But what if instead they were useful, and they were a reference available to simplify life? You don’t read the dictionary, but you know that its there and handy when you need it… so should be process and policy.
I feel passionate about these things because I hate to see employees stressed out because they aren’t sure what to do or how to do something. Useless anxiety. Wasted energy. Muda. I see managers beat on their people for not knowing… but who’s fault is it really? There are hard problems in the world, lets focus the energy on new problems and codify what we’ve learned in the past for everyone to benefit from. This is the nature of continuous improvement… building a collective body of corporate knowledge and continuously expanding, refining, and even replacing it when appropriate.
| Agenda | ||
| 9:30 | Willkommen zum Frühstück | |
| 10:00 | Solaris 11 im Detail - Einbindung in heterogene Netze (CIFS-Dienst etc.) Joerg Moellenkamp |
|
| 11:30 | Kaffeepause | |
| 12:00 | ZFSSA praktisch Einbindung einer ZFS SA in heterogene Netze, aber wie? Vortrag & Livedemo unter VirtualBox Dirk Nitschke |
|
| 13:30 | geplantes Ende | |
You heard me in the past talk about the HCC feature for Oracle databases. Hybrid Columnar Compression is a fantastic, built-in, free feature of Oracle 11Gr2. One used to need an Exadata to make use of it. However, last October, Oracle opened it up and now allows it to work on ANY Oracle DB server running 11Gr2, as long as the storage behind it is a ZFSSA for DNFS, or an Axiom for FC.
If you're not sure why this is so cool or what HCC can do for your Oracle database, please check out this presentation. In it, Art will explain HCC, show you what it does, and give you a great idea why it's such a game-changer for those holding lots of historical DB data.
Did I mention it's free? Click here:
http://hcc.zanghosting.com/hcc-demo-swf.html
While most customers immediately understand how the magic of Oracle's Hybrid Columnar Compression, intelligent storage servers and flash memory make Exadata uniquely powerful against home-grown database systems, some people think that Exalogic is nothing more than a bunch of x86 servers, a storage appliance and an InfiniBand (IB) network, built into a single rack.
After all, isn't this exactly what the High Performance Computing (HPC) world has been doing for decades?
On the surface, this may be true. And some people tried exactly that: They tried to put together their own version of Exalogic, but then they discover there's a lot more to building a system than buying hardware and assembling it together. IT is not Ikea.
Why is that so? Could it be there's more going on behind the scenes than merely putting together a bunch of servers, a storage array and an InfiniBand network into a rack? Let's explore some of the special sauce that makes Exalogic unique and un-copyable, so you can save yourself from your next 6- to 12-month science project that distracts you from doing real work that adds value to your company.
The backbone of Exalogic is its InfiniBand network: 4 times better bandwidth than even 10 Gigabit Ethernet, and only about a tenth of its latency. What a potential for increased scalability and throughput across the middleware and database layers!
But InfiniBand is a beast that needs to be tamed: It is true that Exalogic uses a standard, open-source Open Fabrics Enterprise Distribution (OFED) InfiniBand driver stack. Unfortunately, this software has been developed by the HPC community with fastest speed in mind (which is good) but, despite the name, not many other enterprise-class requirements are included (which is less good).
Here are some of the improvements that Oracle's InfiniBand development team had to add to the OFED stack to make it enterprise-ready, simply because typical HPC users didn't have the need to implement them:
In short: Oracle elevated the OFED InfiniBand stack into an enterprise-class networking infrastructure. Many years and multiple teams of manpower went into the above improvements - this is something you can only get from Oracle, because no other InfiniBand vendor can give you these features across the whole stack!
So let's assume that you somehow were able to get your hands on an enterprise-class IB driver stack. Or maybe you don't care and are just happy with the standard OFED one? Anyway, the next step is to actually leverage that InfiniBand performance. Here are the choices:
While more bandwidth is always a good thing, it's actually the low latency that enables superior performance for your applications when running on any networking infrastructure: The lower the latency, the faster the response travels through the network and the more transactions you can close per second.
The reason why InfiniBand is such a low latency technology is that it gets rid of most if not all of your traditional networking protocol stack: Data is literally beamed from one region of RAM in one server into another region of RAM in another server with no kernel/drivers/UDP/TCP or other networking stack overhead involved!
Which makes option 1 a no-go: Adding TCP/IP on top of InfiniBand is like adding training wheels to your racing bike. It may be ok in the beginning and for development, but it's not quite the performance IB was meant to deliver.
Which only leaves option 2: Integrating your middleware with fast, low-level InfiniBand protocols. And this is what Exalogic's "Exabus" technology is all about. Here are a few Exabus features that help applications leverage the performance of InfiniBand in Exalogic:
As you see, “Exabus” is Oracle's word for describing all the InfiniBand enhancements Oracle put into Exalogic: OFED stack enhancements, protocols for faster IB access, and InfiniBand support and optimizations at the virtualization and middleware level. All working together to deliver the full potential of InfiniBand performance.
Who else has 100% control over their middleware so they can develop their own low-level protocol integration with InfiniBand? Even if you take an open source approach, you're looking at years of development work to create, test and support a whole new networking technology in your middleware!
And then there are the other advantages of Engineered Systems that are true for Exalogic the same as they are for every other Engineered System:
. Less integration pain for you, faster time to market.
For more business-centric values, read The Business Value of Engineered Systems.
And here's the reason why it's not easy to "build your own Exalogic": There's a lot of work required to make such a system fly. In fact, anybody who is starting to "just put together a bunch of servers and an InfiniBand network" is really looking at a 6-12 month science project.
And the outcome is likely to not be very enterprise-class. And it won't have Exalogic's performance either.
Because building an Engineered System is literally rocket science: It takes a lot of time, effort, resources and many iterations of design/test/analyze/fix to build such a system. That's why InfiniBand has been reserved for HPC scientists for such a long time.
And only Oracle can bring the power of InfiniBand in an enterprise-class, ready-to use, pre-integrated version to customers, without the develop/integrate/support pain.
For more details, check the new Exalogic overview white paper which was updated only recently.
P.S.: Thanks to my colleagues Ola, Paul, Don and Andy for helping me put together this article!
For the second time in as many quadrennial dtrace.confs, I was impressed at how well the unconference format worked out. Sharing coffee with the DTrace community, it was great to see some of the oldest friends of DTrace — Jarod Jenson, Stephen O’Grady, Jonathan Adams to name a few — and to put faces to names — Scott Fritchie, Dustin Sallings, Blake Irvin, etc — of the many new additions to the DTrace community. You can see all the slides and videos; these are my thoughts and notes on the day.
Bryan provided a typically eloquent review of the state of the community. DTrace development is alive and well — after a lull while Oracle’s acquisition of Sun settled in — with new support for a variety of languages and runtimes, and new products that rely heavily on DTrace as a secret sauce. Bryan laid out some important development goals, areas where many have started straying from the edges of the completed DTrace features into the partially complete or starkly missing. We all then set to work hammering out a loose schedule for the day; I’ll admit that at first I was worried that we’d have too many listeners and not enough presenters, but the schedule quickly filled — and with more topics than we’d end up having time to cover.
DTrace, from its inception, has been a systemic analysis tool, but the earliest development focused on kernel observability — not a surprise since Bryan, Mike, and I developed it while working in the Solaris kernel development. After its use spread (quickly) beyond the kernel team, use shifted more and more to features focused on understanding C and C++ applications in user-land, and then to applications written in a variety of higher-level languages — Java, Ruby, Perl, Javascript, Erlang, etc. User-land Statically Defined Tracing (USDT) is the DTrace facility that enables rich tracing of higher-level languages. It was a relatively late addition to DTrace (integrated in 2004, well after the initial integration in 2003), and since then we’ve learned a lot about what we got right, what we got wrong, and where it’s rough — in some cases very rough — around the edges.
In his opening remarks, Bryan identified USDT improvements as a key area for the community’s focus. In DTrace development we tried to focus on making the impossible possible rather than making the possible easier. In its current form, some things are still impossible with DTrace, namely consumption of type structures from user-land programs; stable, non-privileged use of DTrace; and support for different runtime versions. Dave Pacheco and I took the first slot on the schedule and spoke (at length — sorry) about solutions to these problems.
While others had the benefit of a bit more time to prepare, I did have the advantage of spending many years idly contemplating the problem space and possible solutions. On the subject of user-land type information (in the form of CTF), I identified the key parts of the code that would would need some work. For the USDT enhancements, we discussed dynamic translators — D code that would be linked and executed at runtime, contrasted with today’s static translators that are compiled into a D program — how they would address the problem, and how these ideas could be extended to the kernel (for once, user-land is actually a bit ahead).
I’ll go into the details of our off the cuff proposals, and delve into the code to firm up those ideas in a future blog post. Beyond the extensive implementation work we laid out, the next step is to gather the most complicated, extant USDT providers and proposals for other providers, and figure out what they should look like in the new, dynamic translator world.
Next up, my long-time colleague, DTrace contributor, Eric Schrock led the discussion on D language additions. The format of a D program is heavily tied to DTrace’s implementation: all clauses must trace a fixed amount of data, and infinite loops are forbidden. For this reason, D lacks the backward branches needed for traditional looping, subroutines for common code, and if/else clauses for control flow. Each of these has a work-alike — unrolled loops, macros, and predicates or the ternary operator — but their absence renders D confusing to some — especially those unaware of the motivation. Further, the D language need not necessarily hold the underlying implementation so central.
Eric discussed some proposals for how each might be addressed, and I noted that it would be possible to create a prototype environment where we could try out these “D++” features by compiling into D work-alikes. The next step is to identify the most complicated D scripts, and see what they might look like for various incarnations of those language features.
The next few sessions focused not on changes to DTrace, but interesting work done using DTrace:
John Thompson of Sony talked about their port of DTrace to the Playstation Vita (!). Sony developers are given access to DTrace, but found it to be unfamiliar and unapproachable. John spoke his attempts to remedy this by replacing D with a C++-like interface which he implemented by replacing the D compiler with Clang.
My Fishworks colleague, Brendan Gregg, showed some of beautiful visualizations they’ve been developing at Joyent, and talked about the analyses those visualizations enabled. As always, it was fascinating stuff. If you don’t read Brendan’s blog, you really should. Long-time DTrace advocate, Theo Schlossnagle, talked about the visualizations they’re doing in Circonus — also fascinating stuff for anyone thinking about how to present system activity in comprehensible ways. Richard Elling showed the DTrace-based visualizations Nexenta used at VMworld to rave reviews.
Mark Cavage presented Joyent’s work bringing DTrace to node.js; Scott Fritchie talked about DTrace for Erlang. Both were useful sources of ideas for how we could improve USDT.
Ryan Stone presented the state of DTrace on FreeBSD. That DTrace is not enabled in the build by default remains a key obstacle for adoption. I hope that Ryan et al. are able to persuade the FreeBSD leadership that their licensing fears are misguided.
I was delighted that Kris van Hees was able to attend to present the Oracle port to Linux. DTrace for OEL was announced at Oracle Open World 2011, but the initial beta didn’t live up to its billing at OOW. As is often the case, this was more a failure of messaging than of engineering. Kris and his team are making steady progress. While it’s not yet in the public beta, they have the kernel function boundary tracing provider (fbt) implemented. Most heartening of all, Oracle intends to keep DTrace for OEL moving forward as the community evolves and improves DTrace — rather than forking it. How that plays out, and what that means for DTrace on Oracle Solaris will be interesting to see, but it’s great to hear that Kris sees the value of DTrace ubiquity and DTrace compatibility.
As was remarked several times, having DTrace available on the fastest growing deployment platform will be the single most significant accelerator for DTrace adoption. The work Kris and his team at Oracle are doing is probably the most important in the DTrace ecosystem, and I think that I speak for the entire DTrace community in offering to assist in any way possible.
Matt Ahrens and George Wilson — respectively the co-inventor of ZFS, and the preeminent SPA developer — presented a proposal for a DTrace provider for ZFS. ZFS is a highly sophisticated filesystem, but one that is also difficult to understand. Building in rich instrumentation is going to be a tremendous step forward for anyone using ZFS (for example, our mutual employer, Delphix).
Jarod Jenson — the first DTrace user outside of Sun — took the stage in the final session to talk about DTrace adoption. Jarod has made DTrace a significant part of his business for many years. What continues to amazing him, despite numerous presentations, demonstrations, and lessons, is the relatively low level of DTrace adoption. DTrace is a tool that comes alive in the hands of a skilled, scientific, incisive practitioner — and in all of those, Jarod is superlative — but it can have a high bar of entry. There were many concrete suggestions for how to improve DTrace adoption. Most of them didn’t hold water for me — different avenue for education, further documentation, community outreach, higher level tools, visualizations, etc. — but two were quite compelling: DTrace for Linux, and DTrace on stackoverflow.com (and the like). I don’t know how much room there is to participate in the former, but by all means if there are DTrace one-liners that solve problems (on Mac OS X for example), post them, and get people covertly using DTrace.
The core DTrace community is growing. It was great to see old friends like Steve Peters who worked on porting DTrace to Mac OS X in the same room as Kris van Hees as he spoke about his port to Linux. It was inspiring to see so many new members of the community, eager to use, build and improve DTrace. And personally it inspired me to get back into the code to finish up some projects I had in flight, and to chart out the course for some of the projects we discussed.
Thanks to everyone who attended dtrace.conf in person or online. And thanks especially to Deirdre Straughan who made it happen.