The Kitware Blog

Open Science and Hurdling The Complexity Barrier

Will Schroeder — Wed, 16 May 2012 09:43:35 -0400

A large software system is one of the most complex machines ever built. When you combine complex software with big data, the number of possible execution paths are inconceivably huge. So how is it that software engineers can build and run these enormously sophisticated machines? (Yes I agree these systems are far from perfect, but in general I've seen significant progress over the many years that I've been in the field, so please bear with me.)

There are several answers to this question, ranging from encapsulation (e.g., object-oriented approaches used to hide complexity behind interfaces), to distributed processing technologies (enabling simple components to interoperate), to good software practices (such as code generators), to test-driven software processes (like the CTest/CDash process); all of which have been well-known for many years, as even a brief search will tell you [Levent] [MacDonald] [Kuhn]. The point is that as software developers, we realize that the technology foundations we build are critical to making future progress and that managing complexity is central to our practice. We know that our systems must generate reproducible results if we are to depend on them; otherwise applications built on them are worthless, and may result in grievous harm in the worst case.

With this generally successful example of managing complexity through reproducibility behind us, what are we to make of the recent dismal report found in Nature which states that on the order of 90% of papers published in science journals describing "landmark" breakthroughs in preclinical cancer research describe work that is not reproducible, and are thus just plain wrong? (C. Glenn Begley, formerly head of cancer research at pharmaceutical company Amgen, and Lee M. Ellis a cancer researcher at the University of Texas, authored this astonishing paper.)

Repeat after me: approximately 90% of paper results were not reproducible. I don't know about you, but this paper result, if true, is one of the most disturbing scientific findings that I have ever read. Just think of what this means to pharmaceutical companies and other healthcare providers who are building therapies based on these suspect results; we are talking possible harm to patients, billions of dollars misspent, and years of potentially wasted work. Mostly it shakes my faith in the foundations of this research, and I wonder how we are to advance the scientific frontier and maintain public trust in the scientific method. And once again I'm reminded of the imperative to practice Open Science.

What's common about medical research and software is that they are both enormously complex undertakings. Being human, there will be errors and biases, and in the worst cases even fraud. The only way to combat these unfortunate human qualities is to provide methods to identify and correct issues. As any open source practitioner and dedicated scientist will tell you, producing reproducible results using transparent methods and data, which are then made publicly available to others, are some of the best ways to do this.

In the past we've gotten away with much. Methods and data were hidden or obfuscated behind opaque papers, firewalls, and claims regarding intellectual rights. Often the relatively low-level of complexity was such that outsiders could work around these barriers by reproducing software or repeating a research trial. But times are different now and the increasing sophistication of our technologies and the cost of scientific research means that we have to pull out all the stops to address the looming Complexity Barrier. The scale of the challenge is such that if we do not, our technological progress will be greatly impeded if not halted. To make progress and build the foundations of technology, the Complexity Barrier demands that we do so by building transparent, open, and reproducible results that can be verified and ultimately relied upon. For some scientific method purists, it could be that this paper which repudiates so many results is simply science in action, and given enough time the scientific method always corrects itself. This could be, but my suspicion is that self-correction is now harder to do, with closed systems contributing much of the burden.

Certainly there are moral, ethical, and philosophical arguments to be made here, which are probably not going to motivate some of us on a day-to-day basis. However, as an engineer-at-heart who likes to get things done and make a difference, there is a very pragmatic rationale for doing business and making a living in accordance with the three Open Science principles of Open Access, Open Data, and Open Source. That is: if you don't practice these principles, it is likely that those who do so will leave you in the dust, whether measured in terms of career accomplishments, innovation impact, or business success. As the common saying "Go Big or Go Home" is used to describe a winning competitive strategy, we now have to mind the corollary "Go Open or Go Home" if we are to compete in the modern world.

In a sadly ironic twist, this important paper is only available through a closed pay-wall. Further, the authors propose that these circumstances are due to the high-pressure research environment that forces researchers to publish or die. Specifically, they say researchers must be more willing to report negative findings in their papers and that research facilities should change their policies regarding publishing. All likely true, but they miss the essential conclusion: science that is not reproducible is not science; we must remove barriers to reproducibility to if we are to hurdle the Complexity Barrier and advance scientific knowledge.

Fortunately some of the most prestigious journals are embracing the practice of reproducible verification, and leading the way towards restoring the practice of scientific research. It is now up to each one of us to support these initiatives and to promote the adoption of Open Science and reproducibility in the technical societies, institutions, and communities in which we participate.

Open Source and the Innovator's Dilemma Revisited

Will Schroeder — Thu, 10 May 2012 11:45:15 -0400

If you have been participating in the free and open source movement as long as I have, you know that the foundational arguments for open source development have been made, and the initial skirmishes fought and won. From my own experience as a software developer, anecdotal evidence, and emerging business studies, it’s clear to me that open source software practices are more agile, scalable, cheaper, higher quality, authentic, and responsive to the development of custom enterprise software. Savvy organizations are realizing this, and open source approaches are making their way into everything from research in the computational sciences, government infrastructure (including the U.S. Department of Defense), and technology integration in forward-thinking companies.

Despite these successes, the struggle is far from over. The disruptive nature of the open process is now keenly felt in the mainstream and enterprises are realizing that without plans to accommodate these disruptive processes, they may soon join the company of Tyrannosaurs Rex. Thus, there is an existential struggle underway between companies that want to sell off-the-shelf solutions, and those that want to collaborate with and serve their customers to create custom solutions. Or stated another way, companies that want to control the customer experience are contending with those that want to enable the customer experience.

As a technology leader in an open source company, I find this state of affairs to be thrilling. I often think back to when I read The Innovator’s Dilemma where the dynamics of disruption are so clearly laid out. What is often forgotten about this book is that not only did it describe disruptions due to new product introductions, but disruptions due to process changes as well. For example, the process change from variety retail to discount retail wiped out Woolworth and eventually brought on the dominance of WalMart.

Will open source do to closed source software what discount retailing did to Woolworth? I doubt it, as Apple Computer has shown that a well-crafted and controlled user experience can be profoundly satisfying (and profitable). On the other hand, even the notoriously closed and controlling Apple has had to “open” itself by enabling the hundreds of thousands of non-Apple apps, and by building on critical open source projects, such as BSD and WebKit/KHTML, to reinvent Apple OS X. So even though open may not dominate outright, the enabling expectations in the user community for openness, freedom, customization, and reasonable pricing will continue to exert significant pressure on vendors who wish to control the customer experience.

I find it interestingly ironic that it is reported that one of Steve Job’s favorite books was The Innovator’s Dilemma; I wonder if he ever imagined an open software process being potentially disruptive to his company.

On the other hand, there are areas in which open approaches will clearly dominate, including the development of custom, large-scale software and performing computational R&D to name a couple. The ability to collaboratively create scalable software that meets the needs of an enterprise, is free of intellectual property barriers, and uses agile software processes is an overwhelming advantage. Further, if the right licensing strategy is used, open source companies can successfully provide services to their customers and make compelling arguments against vendor lock-in and for rapid technology integration and adaption, both of which are increasingly important capabilities desired by tech-savvy companies.

Thus, we are witnessing a profound playing out of the FOSS movement. It’s now penetrating the mainstream and showing that it is much more than a philosophical movement, but rather is a fierce competitor with an important role in innovating and creating business opportunities. It’s going to be quite a show.

University of Albany : Open Source Festival

Luis Ibanez — Thu, 12 Apr 2012 08:18:55 -0400

On Thursday, March 29th, the University of Albany Student Chapter of the American Society for Information Science & Technology (ASIS&T) organized the 2nd Annual:

UAlbany ASIS&T Open Source Festival 2012

https://sites.google.com/site/opensourcefestival2012/

at the Campus Center of SUNY Albany.

Kitware joined the event with the participation of Will Schroeder, Bill Hoffman, and Luis Ibanez who delivered the following talks:

Open Source Business Models - Will Schroeder spoke about the variety of business models that have been used to monetize open-source software and communities. (This was a timely talk given Red Hat's recent announcement that the company exceeded $1 billion in revenue by relying mostly on a subscription model.) Dr. Schroeder discussed a variety of business approaches ranging from open source as a hobby to consulting, dual licensing, and collaborative R&D (which is a major component of the Kitware model).
Open Source Quality Processes - Bill Hoffman discussed how open-source practices result in higher-quality software due to the many eyes theory, which states that the more people with access to source code, the quicker bugs are detected and fixed. Although not all open-source projects will benefit from this theory as much as the Linux kernel, there are open tools and processes that enable developers to share and test software regardless of community size. Tools including Git, CMake, CTest, and CDash are examples of freely-available tools for managing, building, and testing high-quality software – while also reducing maintenance costs and maximizing community involvement.
Open Source in Healthcare - Luis Ibanez introduced the economic relevance of healthcare costs in the US, currently 18% of GDP, and described how the open-source community can contribute in reducing the cost of healthcare and simultaneously increasing its quality by participating on the open-source initiatives surrounding the development of electronic health record (EHR) systems. In particular, in the case of the Veterans Health Information Systems and Technology Architecture (VistA), and the open-source EHR Agent (OSEHRA).
Open Source in Education - Luis Ibanez also presented on how Kitware has partnered with Rensselaer Polytechnic Institute (RPI) since 2007, to offer an Open-Source Software Practices class. The sixth edition of this class is being taught in the current Spring 2012 semester. In this class, students are exposed to basic principles of economics; models of peer-production; social principles of collaboration; the legal basis of copyrights, patents, and trademarks; principles of software licensing; and the motivational aspects of cooperation. Simultaneously, students are able to practice interaction skills with open-source communities through use of revision control systems, mailing lists, forums, wikis, code peer-review systems, software quality control, and documentation.

The event brought together many groups invested in open source, along with others interested in learning more about the Way of the Source.

Some of the highlights of the event included:

Two open-source initiatives of the New York State Senate were presented:

Indexing Legislation with Lucene, by Graylin Kim, described a web-based application for making the content of legislation in process be made available to the general public. The OpenLegislation service enables multifaceted, full text searching across legislation for the 2009 and 2011 sessions. Lucene, an open-source full text search engine, is used to power this service and open the New York Senate's legislative process to the public.
Open Source in the New York State Senate, by Ken Zalewski, presented how in 2009, the New York State Senate began building the next generation of government applications on a platform of open source software. Today our new constituent information system, the nysenate.gov website, and OpenLegislation services run on a variety of open source software packages including: Drupal, CiviCRM, MySQL, Apache2, Tomcat, Java, PHP, Varnish, Lucene, and Memcache.

Dr. Tara Lindsley from Albany Medical College presented on the use of open-source software for the creation of a Virtual Brain Model, to be used as a learning resource in neuro-anatomy classes. One of the most challenging learning objectives in the medical neuroscience curriculum is to understand the complex spatial relationships of functionally-important structures buried deep inside the brain, especially those that are too small to resolve on an MRI. This presentation described how a team of students and educators at Albany Medical College used open-source software as a platform to create a program that allows the user to build 3D virtual models of the human brain, showing up to 80 different structures. The models can be zoomed, rotated 360° and varied in color and opacity, then saved or printed as study aids. The software is available at: http://www.amc.edu/academic/software/, and is based on ITK and ITK-SNAP.

So You Want to Open Source Your Code?

Will Schroeder — Wed, 11 Apr 2012 10:05:41 -0400

I'm happy to say that here at Kitware we are seeing a strong, recent uptick of interest for "open sourcing" existing closed code. This interest is emerging across the board from research organizations, academics, national laboratories, government agencies, funding agencies and commercial enterprises. It's totally consistent with the "open wave" embracing all of us, as mainstream organizations recognize the many benefits of open source--cost savings, agile innovation, avoidance of vendor lock-in, increased software quality, distributed maintenance burden, leveraged community expertise, and so on. Despite this happy turn of events there is need to proceed cautiously: it's not enough to dump some code into a repository, write a README document, and open the Internet floodgates. In this blog I'll outline some of the steps that I think are necessary to successfully open source existing software. My motives for this are simple, I don't want to see open source tainted by bad practices, or see contributing organizations turned off by a bad experience.

Get your IP ducks in order. This is pretty obvious, but make sure that the ownership of the code is clear, and that you or your organization has the right to license software under new (open source compatible) terms.
Use the right license. There are many possible OSI-compliant licenses available, choose the right one consistent with the goals of the project. For example, if you expect to collaborate with businesses and want minimal barriers to sharing, use something like an Apache or BSD license (this allows you to more easily work in environments with mixed open and proprietary code). GPL (and other reciprocal licenses) has its place too, but it can often scare collaborators away.
Establish a software process. From day one the system should have a formal software process in place including a repository; build, test, deploy and documentation infrastructure; a bug tracker; and organizational documents (e.g., coding style guide, requirements, and so on). There are many tools for implementing such a process, we prefer using git (repository), gerrit (code review), CMake (build), CTest and CDash (testing), and CPack (packaging). We often use Doxygen (documentation) and Mantis (bug tracking); other great choices are available too.
Define methods for community interaction. Along with the software process, particular attention should be given to building and supporting the community. Communication channels should be defined (mailing lists, wikis, web pages, periodic phone calls, periodic face-to-face code-a-thons). Governance is important too, although there is a strong dependence on the size of the community; smaller communities may be fairly ad-hoc, larger ones may require formal governing boards and a hierarchy of lieutenants to govern and manage them. Also, make sure that the transition from an autocratic closed-source situation to an open source community leaves behind any politics and inefficient organizational hierarchies.
Nurture the community. Community does not happen spontaneously, it requires a lot of daily work and persistence. Newcomers to open source tend to be too focused on software technology and miss the fundamental importance of community. If one has to choose between investing efforts in the community or technology, community should always come first, given that a healthy community will necessarily take care of the technology, while technology will not spur the emergence of a community. We have seen large-scale, well-funded projects shutdown because they were unable to build a community around the software. Pay attention to educating the next generation, welcoming strangers, building culture, improving skills, and creating space for professional and personal growth. A community should be a place where members feel at home, and where they grow and feel appreciated.
Insist on high emotional IQ. Many open source projects fail because their communities are toxic. Oldsters berate newcomers for asking "stupid" questions; flame wars erupt; antagonistic and divisive behavior turns what should be a pleasurable activity into an ordeal, and so on. By fostering positive relationship practices (recognizing contributions, providing encouragement, carefully listening to and considering other's ideas), it's amazing how fulfilling participating in a community can be, and how much fun!
Provide experts with the code. Software is a codification of human knowledge; the knowledge of experts who understand an application area in a very deep way. Releasing code into the wild without experts to nurture it is a bad idea. After all we teach geese to migrate south, similarly captive closed software needs help to fly again. So find a way to keep the code experts involved, even if only on a consulting basis. And to belabor the point, experts are most effective when they nurture and mentor the community. For example in the VTK community, we are fortunate to have two retired experts (Bill Lorensen and Andrew Maclean) who are essential, if not critical, parts of the community.
Include open data. Don't forget to contribute challenging and representative data to the project. Without the proper data it's almost impossible to properly test and advance the code base. It is essential that data is gathered and curated just as the code base is.
Build a sustainable business model. While many communities do survive on volunteer effort, it is important to consider how future development is to be paid for. To some extent this is a matter of scale and complexity; smaller communities requiring deep expertise and significant time investments may require funding from external agencies or support from donors. Larger communities may self-organize into efficient volunteer organizations; but even then finding resources for advocacy, resources and collaboration (like conferences) can be very important. Business models built around services and technology integration are a natural fit to the open source enterprise.
Work with open source experts. Work with companies like Kitware and open source developers who have had the experience of creating, growing and maintaining open source communities. These experts breathe collaboration and know how to foster communities.

Notice that in list above I did not suggest technical changes to the software like "re-architect the code" or "rework the API" or "document more" or "rewrite for common style". Obviously these are all great things to do prior to open sourcing a project, but they often serve as excuses and delays to getting the code out. In practice, these technical details will be taken care of automatically as soon as you build a healthy community around your project. There is a balance between perfection and getting stuff done. To use a crude analogy that makes our HR department cringe: you can't be afraid to show your underwear. Open source means revealing more than you may want to; get over it, it's the nature of the beast and essential to getting the community involved. As counter-intuitive as it may seem, we have found that by embracing imperfection open source communities address deficiencies at a faster rate than most closed systems, resulting in high-quality software. So get the code out; after all if the code is never released, it's never open sourced.

Open sourcing an existing code base is a wonderful step to take if done with the community, quality and sustainability in mind. To my way of thinking it's a great way to help lay the foundation for accelerating computational innovation. And remember, open source is all about collaboration, so if you are uncertain about how to go about it, ask for help and contact Kitware, we are the open source collaboration experts.

Surviving the Valley of Death

Will Schroeder — Mon, 12 Mar 2012 10:42:11 -0400

Search the web for “technology transition” and “valley of death” and you will run headfirst into an ongoing concern with the process of commercializing basic research (here’s one such report A Valley of Death in the Innovation Sequence: An Economic Investigation). As this report (and many other sources) characterize it, the Valley of Death in the innovation process occurs due to a “dearth of sources of funding for technology projects that no longer count as basic research but are not yet far enough along to form the basis for a business plan.” The result is that innovative technologies often go to the Valley to die a lingering death due to lack of bridge funding in the transition from idea to practice.

Over the years, various programs and initiatives have been created to address this challenge; for example, the excellent US Government SBIR/STTR program has been specifically designed to support the technology transition process. While this and other initiatives are certainly making a difference, in my opinion the Valley is growing wider and deeper with each passing year. Governments, in a worldwide struggle in which R&D is viewed as a competitive advantage, are funding ever more research, and a flood of money is arriving from philanthropists and non-profits (for example, in medical research) to push our basic scientific understanding in a diverse variety of fields including medicine, biology, materials, energy, computing systems, and many more. However, the gap remains because we are too focused on the discoveries and not the practical application of knowledge, which is where the impacts are really made. Many philanthropists are much more interested in discovering the cure for cancer than developing systems to deploy existing knowledge. Yet this is precisely where positive differences in the welfare of people are made most often.

My take is that the balance between the so-called three pillars of science -- experiment, theory, and computation -- is dramatically shifting in favor of much more computation; yet we have not manifested the corresponding processes to support the transition of computational discoveries nor maintain valuable resources (including software, data, publications and related services--see Data-Driven Science: A New Paradigm?). Many of today's research projects do not recognize the scale to which data and science is driving innovation, and do not fully appreciate the need for skilled computational scientists, and evolved software and data processes to make the discoveries real.

In my experience, some in the scientific community view computation as a second-class tool to confirm theories, augment experiments, or process data; have a primitive understanding of software and the computational process; and do not recognize that the future of science is computation as the driving force behind discovery. As a result, there is an imbalance in funding and in the way research programs are organized. In many labs there is just enough research money to get the job done (e.g., publish a paper), but the behind-the-scenes software and data situation is a mess, meaning that the potential to transition the software to practice is negatively and significantly impacted. Thus, widening occurs as we push the "knowledge" side of the Valley away from the "practice" side.

Here's a typical example I often see. When visiting a prestigious academic research group, a non-profit research organization, a commercial research center or even a national laboratory, I am often impressed and amazed at the work going on, but find myself completely aghast at the software processes (if any), data management practices, and even publication process. In many cases, source code is not held in a repository (every student has their own slightly-modified version); data, which costs millions of dollars to acquire, is placed on portable hard drives and stashed in filing cabinets; and complex computational processes are documented without regard to the principles of Open Access. Moreover, unreasonable IP barriers often rear their ugly head as code is viewed as proprietary or built on top of closed systems. All these and more make the technology transition process that much harder.

What to do about this? Start with openness, build a community, engage proficient software developers, and remove barriers to collaboration. Engage with companies like Kitware who have the smarts to engage in computational science, yet with the passion to build maintainable, reusable, and extensible software systems, processes and repositories. Hire more computational scientists and software people for the lab and treat them as first class citizens of the research team. Balance funding portfolios to recognize the increasing importance of computation in the innovation process. My belief is that by taking these steps to reduce the depth and breadth of the Valley, scientists will move their knowledge to practice much faster, make impacts much sooner, and manifest the practical applications of knowledge to receive accolades and future funding.

Open Source Funding Streams

Will Schroeder — Tue, 27 Dec 2011 08:43:00 -0500

During the holiday season it's natural to consider giving back and reflecting on what we've accomplished. According to ohloh.net the open source community has done an amazing job on both counts. For example, a partial tabulation of some open source projects related to medical image analysis yields a total of over $350 million and over 12 million lines of code, the result of years of hard work .

WOW!

Do I believe these numbers? Absolutely not. There are many obvious problems: COCOMO cost estimation is notoriously inaccurate; many of our OS developers are way more productive than the model assumes; line counts are off; and the cost of a developer ($100K/year assumed) varies widely (probably factors of 0.5 to 3 depending on the organization). On the other hand, many software tools are not included, for example ParaView is often used in medical applications, so this is far from a complete accounting. It would be really fun to do a financial roll up of open source tools being used, in say medical research, and compare them to investments made by funding agencies, research organizations and other customers. I have a pretty strong suspicion that a dollar invested pays off very handsomely.

Whatever the real numbers, I do believe they are impressively large, hence implicit in these figures is that many of these projects represent mammoth combinations of money, vision and concerted effort over decades (the first line of VTK code was written in late 1993) to create ongoing funding streams. I attribute much of this success story to the Unsung Heroes of Open Source, those champions with a clear Collaborative Vision of the future who are willing to support projects over the long haul. However, while having Champions and articulating Vision are essential to success of an open source project, to manifest portfolios of this size it's important to understand the creative energy that goes into supporting them over the long term. Thus in this blog post I'd like to describe a few of the many approaches our communities have used to create successful funding streams, as well as reflecting on the challenges that the various approaches present. Hopefully in the coming years we can do more of the same and approach the billion dollar mark.

Community Efforts: The classic model of open source development as being volunteer driven is giving away to more sophisticated approaches which combine volunteers, students, academics, research labs, non-profits and commercial firms. In our communities we leverage these partnerships in novel ways by hosting hackfests to bring together developers and users in concentrated programming events; teach college-level open source courses and harness the energy of students; build open source research programs such as NA-MIC which sponsor major Project Weeks (summer and winter) to address algorithm and software development needs; get the word out at important conferences (Supercomputing, VisWeek, OSCON for example); participate in broader community events like Google summer of code; and sponsor community-building webinars. The challenge here is to establish effective governance and communication channels in order to effectively weave these efforts together.

Research Programs: There is an avalanche of formal support for Open Access, Open Data and Open Source research initiatives. Despite inherent resistance to some in the scientific community to sharing (for a discussion see the excellent book Reinventing Discovery: The New Era of Networked Science), the Unsung Heroes who manage the research funding stream see the bigger picture: Open Science requires funding steams dedicated to making the results of research open. Such research programs represent a significant change in the way R&D is performed, positively impacting many open source communities with funding, and explicitly prioritizing agile innovation and reproducible science as a broader public good. There is also a pragmatic focus to many programs which aim to create open computational infrastructures that spawn competitive commercial activities to create jobs and build businesses.

SBIR/STTR: Businesses like Kitware that engage in R&D may tap the SBIR/STTR program. The Small Business Innovation Research program is a highly competitive program that encourages US small businesses to engage in Federal R&D with the potential for commercialization. While the program is extremely competitive, it has enabled Kitware to develop core infrastructure in several of our open source projects. In the past the program emphasis has been squarely focused on business models that create and commercialize IP (patents, licensing, etc.), and the open source service model was viewed with skepticism. This is now changing as successful open source companies emerge, agile innovation is recognized as a competitive advantage, and broad scientific impacts are made. Other countries or groups offer similar commercially-oriented R&D programs too (and which our foreign offices participate in).

Commercial Work: Many commercial forms work in the background (often choosing anonymity for competitive reasons) creating systems using open source software, and then contribute back everything from bug fixes to small enhancements to major chunks of code. In some cases our work with commercial entities is small potato-stuff like support contracts, and sometimes major technology-integration initiatives valued at millions of dollars. We find that permissive licenses such as BSD work best, since various flavors of GPL (and other reciprocal licenses) are so complex and onerous that commercial firms stay away from them. We've found that even receiving partial contributions from commercial firms (who admittedly often hold back code when not required to by license) produces more code contributed, and more overall support to our communities, than zero code and support due to scary licensing provisions.

Engaging with Non-Profits. Many non-profits have a social mission and are happy to support open source communities (ironically many universities resist open approaches, employing active patent and licensing offices). While non-profits may not contribute direct funding (although some foundations are huge and directly do so), the exposure to the community can be important, not to mention the driving technical challenges. For example, the Give-A-Scan project developed by the Lung Cancer Alliance and Kitware, uses many open source tools to create an archive of images and clinical data from donations by lung cancer patients. We have received favorable press which goes a long way to emphasizing our commitment to creating shared value.

Government. Aside from scientific research programs, governments are recognizing the value proposition of using open source software. Licensing issues, which are extremely difficult to manage across large organization are greatly reduced, and favorable competitive situations are established (firms compete on who services an open source system, and there is little if any vendor lock-in). Moreover, if the government desires the power to change and adapt the underlying software (for example for defense purposes) open source tools are essential. Some of these projects can be huge and the politics is challenging, for example Kitware is part of the OSEHRA team to improve and maintain EHR information systems including the VA's successful Vista system. Despite the challenge of large government projects, such programs can provide significant support to the open source community.

Creating and maintaining effective, long-term funding streams is critical to the success of large scale open source projects. While we've done an amazing job of it thus far, changing the way the word innovates and shares will require that we continue to push the frontiers of creative business development, as well as creating the best possible technology for our customers and collaborators.

Software Forethought

Will Schroeder — Thu, 03 Nov 2011 20:56:00 -0400

Here's an all too common scenario. A bunch of really smart scientists and medical researchers get together. They envision a research program of unprecedented scale. They obtain funding, tens to hundreds of millions of dollars, from academic, commercial, non-profit enterprises, and from investors and philanthropists. The plans are drawn up, brick and mortar is under design and soon to be built! And at some point the data starts flooding in.....

Oops there's a problem, what to do about the data, how do we maintain provenance, analyze, visualize, and share it? How are we going build reproducible software systems and translate our research to application? Well let's just hire some software folks, they'll take care of it, we'll provide computing budgets for the scientists with which to buy software, we've budgeted a cluster (what more do you want after all?), we might even hire a computer scientist or two! In the mean time some of the more tech savvy scientists (never formally trained in software process) will start writing some code. And so it goes, we muddle along.

Just another case of software as an afterthought. I've seen this scenario in action much too often across groups ranging in scale from small research teams to large research institutions. To be blunt (and gentler then many deserve) the results are predictably poor: primitive computing and visualization capabilities, lost data, fractured and incompatible workflows, non existent software processes, inefficient collaboration methods, and poor science. It makes you want to cry for the waste of talent and resources, not to mention the missed scientific discoveries and the possibility of better health care outcomes.

I think it's time to make software cosiderations a forethought, a fundamental driver of the scientific process. I'm convinced doing so will unleash a torrent of innovation. For a long time the scientific process, consisting of theory, experiment and computing (and maybe a data-intensive scientific discovery if you ascribe to the fourth paradigm) has been driven by the experimentalists and theorists, and computing has gone along for the ride. But more and more science is computationally driven, yet I don't think we've reflected this in our thinking, and more importantly in the way we do science. It seems pretty clear to me: it's time to place software and computing front and center in the practice of science, and then work with the theorists and experimentalists to solve their problems using the full potentail of computing technology.

You probably think I'm making this stuff up (I'm not) and being overly dramatic (I am, but it's a blog after all)! For example, today I attended the dazzling, public announcement of the New York Genome Center. This $125 million venture allies some of the most brilliant minds in the genomics field, along with leading commercial, academic and research institutions. It is truly amazing, exciting and inspiring to see this come together, and I expect someday Kitware will be a part of it (after all it's in our New York back yard and they desparately need us, they just don't know it yet).

However as a scientific computing professional I can't help but be concerned. How many software/computing institutions are founders of this Center? How many computational scientists are in leadership positions (as indicated from the biographies handed out at the event)? Zero and zero. More worrying, during the presentations there were all the tell-tale signs of software as an afterthought. Phrases referring to software as a challenge and a limiting concern; vague statements about data sharing plans; and a singular lack of recognition that to a large degree, understanding the genome is a computing problem. As a computational scientist sensitized to these code words, they are strong signals that software will come late to the party (once the data comes flooding in). It could be that there is much going on behind the scenes, everything is under control, and I'm wrong (again) about this, but I've seen software treated as an afterthought too often to not get my spider senses tingling.

As data becomes larger and our modeling and understanding of systems more complex, the computing problem grows in importance. The point is fast approaching (that is, if we are not there already) where the innovation bottleneck will be computing, in particular software. Thus without progress on this front, the worst case scenario is that adding more theory and experimentation will do little to advance science and the medical arts. To make progress we need to break through the computing bottleneck by treating it as the serious challenge that it is.

My recommendation is that we have to think about software proactively. If you are forming a research enterprise, include a software computing professional in a leadership position, or look around and add one to your board. Alternatively engage scientific computing organizations in which software and big data are core competencies. With the right mix of people, organizations, and considered forethought, this Center and many other research initiatives like it can succeed beyond our wildest imaginings.

V&V As a Way of Life

Will Schroeder — Tue, 01 Nov 2011 10:10:47 -0400

There's nothing like a good conference to remind you of the basics.

I recently attended IEEE VisWeek and participated in the panel "Verification in Visualization: Building a Common Culture". The panel was organized by Mike Kirby (Utah) and Claudio Silva (NYU Poly) with Robert Laramee (U. of Swansea) and myself as additional panelists. The focus of this particular panel was on the principles and implementation of validation and verification (V&V) in the computational sciences, with emphasis on the visualization process.

For those of you who don't know, V&V in this context is about ensuring that a software system meets specifications and that it fulfills its intended purpose. Validation addresses the question "Are we building the right thing?" and verification addresses the question "Are we building it right?"

Well who's going to disagree with a panel that advocates a culture of V&V? It's hard to have a provocative and entertaining panel when the thesis is generally agreed on by everyone. But that's one of the key points of the panel: we often take things for granted, so we have to build into our DNA essential, basic principles such as validation and verification to achieve the culture we desire.

My co-panelists, being the talented academics they are, provided very compelling examples of why V&V is critical to algorithm design and implementation. They even had detailed charts indicating how validation and verification should be incorporated into the computational sciences workflow.

Here's where I think some controversy comes in, it's how you practice V&V. Is the process a separate, discrete step as part of the creation and implementation of algorithms (with maybe an occasional V&V workshop thrown in)? Or is it a Way of Life? Meaning do we create an algorithm, make sure it's doing what's it's supposed to do (validate), verify on our sample data sets, and then throw it over the fence (publish) and call it done? Or do we take the approach that validation and verification is an ongoing process that is never really done, consequently requiring us to practice it on a continual basis?

These questions remind me of three related corollaries you often hear in the open source communities we participate in:

If it's not reproducible, it's not science.
If it's not verified, it's not engineered.
If it's not tested, it's broken.

which are all commitments to process. In the open source world, our DNA is ongoing collaboration, evolution, testing and refinement. We know that innovation never ceases, and that continual processes are necessary to stay relevant and vital. Thus V&V is implicitly bred into the bones of open source processes.

So you know where I stand on this: as a practitioner of The Way of the Source, I am a firm believer that verification and validation is a Way of Life, a continual process in which we ask "is our software doing what our users want" and "are we implementing this software correctly?" From hard experience software professionals understand that systems inherently have many dependencies ranging from hardware, to drivers, to the operating system, to system libraries, to the algorithms themselves, and that these dependencies are constantly changing, hence software needs continual testing. We also know that technology changes, and hence our use of technology, consequently a "valid" and effective software system may become invalid, and we may have to revamp or recreate a solution to meet the needs of our users.

If we are to embrace a culture of validation and verification as the panel supposes and to which we all blithely agree, then we need to do the hard work of establishing formal software testing process in combination with continual reevaluation of software effectiveness. Furthermore, because the process is by definition a public certification of the validity and veracity of our systems, open source practices are necessary to support the required transparency and essential evaluation and collaboration processes. It is only when we manifest these practices in our daily software workflow that we can truly say we have established a culture of V&V.

Kitware's Commitment to Open Source Values

Will Schroeder — Wed, 21 Sep 2011 12:53:28 -0400

As many of you know, Kitware is one of the few international, thriving software companies who have made a commitment to open source practices, both in our approach to technology as well as in our business model. We have received recognition from Inc. magazine as being one the fastest growing companies four years in a row. We have sustained an average growth rate of 50% from 2008-2010. We have now broken the 100 employee barrier and are approaching $25 million in revenue. We are making significant impacts on scientific computing and software process. And all of this despite our early concerns about the viability of our approach, which thankfully seem to be unfounded as we are thriving beyond what any of us expected.

And so, after more than 13 years in business, it recently dawned on us that maybe we are doing something right, and others might be interested in how we've done it. To that end, we've been working on a succinct, single page position statement describing Kitware's Open Source Mission Statement.

While the mission statement says it very well, there are a couple of points worth reiterating. We are a service company at heart, so we enjoy collaborating with our customers and communities. We are flexible, therefore we enjoy working with our partners to integrate technology into their workflow, providing both open and proprietary solutions. Finally, we use permissive, simple open-source licenses; we believe that the best way to advance the Way of the Source is to make it ubiquitous and easy to use. We want to get to the point where using and creating open source software is as natural as breathing, and closed solutions seem slow, clumsy and antiquated in comparison. Then we'll know for sure we're doing something right.

How May We Serve You?

Will Schroeder — Tue, 06 Sep 2011 10:34:29 -0400

In our never ending pursuit of the perfect description of the open source business model we've developed some analogies that resonate with many of our customers and collaborators. In particular there are two that seem to cause light bulbs to go off and lots of head nodding to occur. These revolve around tried and true professional services models, tax and legal assistance, two large industries with which we are all familiar. Plus, if these analogies hold true, they lead us to believe that we are in line to receive lucrative TV and movie deals if we play our cards right :-)

Let's start with the US Federal Tax Code: just how big is it? Well no one is completely sure but it appears to be over 3.5 million words and something like 72,000 pages long. Of course nobody actually understands the whole thing, so a large service industry, measured in billions of dollars, has emerged to help prepare tax returns and provide tax strategy advice. And by the way, the tax code is "open source" in that anyone can read it and prepare their own taxes, theoretically we don't need specialists (or for that matter specialists' knowledge embedded in tax-return software) to actually use the tax code. Giving away the Tax Code doesn't seem to have damaged the tax preparation industry; if anything it's given rise to many jobs and thriving businesses.

Similarly, the legal profession is the second largest professional service industry (after health care). Consider the basis for the industry in the US: the Federal Legal Code embodied in ~50 "titles" composed of many volumes each (this doesn't even count the US State Codes). It is open source (for example see the Legal Information Institute which freely provides the US Federal code as well as state and international legal codes). And as many of you know from watching the TV shows Law and Order or Boston Legal, there are a lot of high-paid lawyers running around interpreting, applying and developing these codes (estimate are in the order of a couple hundred billion dollars for the legal services market in the US). In fact, in researching this blog I was taken aback by the way some lawyers describe themselves, it almost sounds like software developers speaking. From the noted attorney Karl Llewllyn in the ABA Journal in 1942, "the essence of our craftsmanship lies in skills and wisdom; in practical, effective, persuasive, inventive skills for getting things done.... We are the trouble-shooters."

Open source communities are similar in that we are creating tens or hundreds of millions of lines of software code (i.e., very large and complex bodies of work, although rooted in science and engineering practices, and built with the aid of testing so the result is not as arcane and inconsistent as the tax and legal codes). In theory anybody can use this body of work for personal or commercial gain. The reality is that it's a lot cheaper, and far more likely to produce better results if you hire the experts. Typically our customers contact us because they have pressing technology challenges, or want to jump their competition by integrating the latest technology into their workflow. Our experts make this happen fast and effectively.

One striking difference is that the tax and legal codes are developed through legislative process, here in the US by representatives of the people they democratically serve. Unfortunately we do not have direct write access to these codes (wouldn't it be interesting if we did?) and must modify the code indirectly through these representatives. In the open source world, public codes are directly created by volunteers, employees, non-profits, academicians and companies who contribute to the public good. However the end result is the same in this sense: what's created is a complex, large knowledge corpus which requires significant expertise to use effectively. Voilà a lucrative service model is born! Of course at Kitware our business model is more complex than this because the service model is paired with our collaborative R&D talents, thus the model combines technology creation with follow on servicing of the technology. This sounds like a virtuous cycle to me and is one of the major reasons why Kitware is such a great place to work.

And this has me thinking, if the lawyers have their own TV shows, why can't we? Something like "Dashboards Gone Red" or "Extreme Debugging". Whether you're interested in this opportunity or more likely, working with some of the finest scientific computing professionals to be found anywhere, contact us and we'll be happy to serve you.