These are the news items I've curated in my monitoring of the API space that have some relevance to the API definition conversation and I wanted to include in my research. I'm using all of these links to better understand how the space is testing their APIs, going beyond just monitoring and understand the details of each request and response.19 Jul 2017
While I track on what is going on with visualizations generated from data, I haven’t seen much when it comes to API driven visualizations, or specifically visualization about API infrastructure, that is new and interesting. This week I came across an interesting example in a post from Netsil about mapping microservices so that you can monitor them. They are a pretty basic visualization of each database, API, and DNS element for your stack, but it does provide solid example of visualizing not just the deployment of database and API resources, but also DNS, and other protocols in your stack.
Netsil microservices visualization is focused on monitoring, but I can see this type of visualization also being applied to design, deployment, management, logging, testing, and any other stop along the API lifecycle. I can see API lifecycle visualization tooling like this becoming more common place, and play more of a role in making API infrastructure more observable. Visualizations are an important of the storytelling around API operations that moves things from just IT and dev team monitoring, making it more observable by all stakeholders.
I’m glad to see service providers moving the needle with helping visualize API infrastructure. I’d like to see more embeddable solutions deployed to Github emerge as part of API life cycle monitoring. I’d like to see what full life cycle solutions are possible when it comes to my partners like deployment visualizations from Tyk and Dreamfactory APIs, and management visualizations with 3Scale APIs, and monitoring and testing visualizations using Runscope. I’ll play around with pulling data from these provides, and publishing to Github as YAML, which I can then easily make available as JSON or CSV for use in some basic visualizations.
If you think about it, thee really should be a wealth of open source dashboard visualizations that could be embedded on any public or private Github repository, for every API service provider out there. API providers should be able to easily map out their API infrastructure, using any of the API service providers they are using already using to operate their APIs. Think of some of the embeddable API status pages we see out there already, and what Netsil is offering for mapping out infrastructure, but something for ever stop along the API life cycle, helping deliver visualizations of API infrastructure no matter which stop you find yourself at.
I am finally getting the time to invest more into the rest of my API industry guides, which involves deep dives into core areas of my research like API definitions, design, and now deployment. The outline for my API deployment research has begun to come into focus and looks like it will rival my API management research in size.
With this release, I am looking to help onboard some of my less technical readers with API deployment. Not the technical details, but the big picture, so I wanted to start with some simple questions, to help prime the discussion around API development.
- Where? - Where are APIs being deployed. On-premise, and in the clouds. Traditional website hosting, and even containerized and serverless API deployment.
- How? - What technologies are being used to deploy APIs? From using spreadsheets, document and file stores, or the central database. Also thinking smaller with microservices, containes, and serverless.
- Who? - Who will be doing the deployment? Of course, IT and developers groups will be leading the charge, but increasingly business users are leveraging new solutions to play a significant role in how APIs are deployed.
The Role Of API Definitions While not every deployment will be auto-generated using an API definition like OpenAPI, API definitions are increasingly playing a lead role as the contract that doesn’t just deploy an API, but sets the stage for API documentation, testing, monitoring, and a number of other stops along the API lifecycle. I want to make sure to point out in my API deployment research that API definitions aren’t just overlapping with deploying APIs, they are essential to connect API deployments with the rest of the API lifecycle.
Using Open Source Frameworks Early on in this research guide I am focusing on the most common way for developers to deploy an API, using an open source API framework. This is how I deploy my APIs, and there are an increasing number of open source API frameworks available out there, in a variety of programming languages. In this round I am taking the time to highlight at least six separate frameworks in the top programming languages where I am seeing sustained deployment of APIs using a framework. I don’t take a stance on any single API framework, but I do keep an eye on which ones are still active, and enjoying usag bey developers.
Deployment In The Cloud After frameworks, I am making sure to highlight some of the leading approaches to deploying APIs in the cloud, going beyond just a server and framework, and leveraging the next generation of API deployment service providers. I want to make sure that both developers and business users know that there are a growing number of service providers who are willing to assist with deployment, and with some of them, no coding is even necessary. While I still like hand-rolling my APIs using my peferred framework, when it comes to some simpler, more utility APIs, I prefer offloading the heavy lifting to a cloud service, and save me the time getting my hands dirty.
Essential Ingredients for Deployment Whether in the cloud, on-premise, or even on device and even the network, there are some essential ingredients to deploying APIs. In my API deployment guide I wanted to make sure and spend some time focusing on the essential ingredients every API provider will have to think about.
-Compute - The base ingredient for any API, providing the compute under the hood. Whether its baremetal, cloud instances, or serverless, you will need a consistent compute strategy to deploy APIs at any scale. -Storage - Next, I want to make sure my readers are thinking about a comprehensive storage strategy that spans all API operations, and hopefully multiple locations and providers. -DNS - Then I spend some time focusing on the frontline of API deployment–DNS. In todays online environment DNS is more than just addressing for APIs, it is also security. -Encryption - I also make sure encryption is baked in to all API deployment by default in both transit, and storage.
Some Of The Motivations Behind Deploying APIs In previous API deployment guides I usually just listed the services, tools, and other resources I had been aggregating as part of my monitoring of the API space. Slowly I have begun to organize these into a variety of buckets that help speak to many of the motivations I encounter when it comes to deploying APIs. While not a perfect way to look at API deployment, it helps me thinking about the many reasons people are deploying APIs, and craft a narrative, and provide a guide for others to follow, that is potentially aligned with their own motivations.
- Geographic - Thinking about the increasing pressure to deploy APIs in specific geographic regions, leveraging the expansion of the leading cloud providers.
- Virtualization - Considering the fact that not all APIs are meant for production and there is a lot to be learned when it comes to mocking and virtualizing APIs.
- Data - Looking at the simplest of Create, Read, Update, and Delete (CRUD) APIs, and how data is being made more accessible by deploying APIs.
- Database - Also looking at how APIs are beign deployed from relational, noSQL, and other data sources–providing the most common way for APIs to be deployed.
- Spreadsheet - I wanted to make sure and not overlook the ability to deploy APIs directly from a spreadsheet making APIs are within reach of business users.
- Search - Looking at how document and content stores are being indexed and made searchable, browsable, and accessible using APIs.
- Scraping - Another often overlooked way of deploying an API, from the scraped content of other sites–an approach that is alive and well.
- Proxy - Evolving beyond early gateways, using a proxy is still a valid way to deploy an API from existing services.
- Rogue - I also wanted to think more about some of the rogue API deployments I’ve seen out there, where passionate developers reverse engineer mobile apps to deploy a rogue API.
- Microservices - Microservices has provided an interesting motivation for deploying APIs–one that potentially can provide small, very useful and focused API deployments.
- Containers - One of the evolutions in compute that has helped drive the microservices conversation is the containerization of everything, something that compliments the world of APis very well.
- Serverless - Augmenting the microservices and container conversation, serverless is motivating many to think differently about how APIs are being deployed.
- Real Time - Thinking briefly about real time approaches to APIs, something I will be expanding on in future releases, and thinking more about HTTP/2 and evented approaches to API deployment.
- Devices - Considering how APis are beign deployed on device, when it comes to Internet of Things, industrial deployments, as well as even at the network level.
- Marketplaces - Thinking about the role API marketplaces like Mashape (now RapidAPI) play in the decision to deploy APIs, and how other cloud providers like AWS, Google, and Azure will play in this discussion.
- Webhooks - Thinking of API deployment as a two way street. Adding webhooks into the discussion and making sure we are thinking about how webhooks can alleviate the load on APIs, and push data and content to external locations.
- Orchestration - Considering the impact of continous integration and deployment on API deploy specifically, and looking at it through the lens of the API lifecycle.
I feel like API deployment is still all over the place. The mandate for API management was much better articulated by API service providers like Mashery, 3Scale, and Apigee. Nobody has taken the lead when it came to API deployment. Service providers like DreamFactory and Restlet have kicked ass when it comes to not just API management, but making sure API deployment was also part of the puzzle. Newer API service providers like Tyk are also pusing the envelope, but I still don’t have the number of API deployment providers I’d like, when it comes to referring my readers. It isn’t a coincidence that DreamFactory, Restlet, and Tyk are API Evangelist partners, it is because they have the services I want to be able to recommend to my readers.
This is the first time I have felt like my API deployment research has been in any sort of focus. I carved this layer of my research of my API management research some years ago, but I really couldn’t articulate it very well beyond just open source frameworks, and the emerging cloud service providers. After I publish this edition of my API deployment guide I’m going to spend some time in the 17 areas of my research listed above. All these areas are heavily focused on API deployment, but I also think they are all worth looking at individually, so that I can better understand where they also intersect with other areas like management, testing, monitoring, security, and other stops along the API lifecycle.
I have self-sensored stories about microservices, because I have felt the topic is as charged as linked data, REST, and some parts of the hypermedia discussion. Meaning there are many champions of the approach, who insist on telling me, and other folks how WRONG they are in the approach, as opposed to helping us work through our understanding of exactly what is microservices, and how to do well.
For me, when I come across tech layers that feel like this, they are something that is very tech saturated, with the usual cadre of tech bros leading the charge, often on behalf of a specific vendor, or specific set of vendor solutions. Even with this reality, I've read a number of very smart posts, and white papers on microservices in the last year, outlining various approaches to designing, engineering, and orchestrating your business, using the "microservices diet".
Much of what I read nails much of the technology in some very logical, and sensible ways--crafted by some people with mad skills when it comes to making sense of very large companies, and software ecosystems. Even with all of this lofty thinking, I'm seeing one common element missing in most of the microservices approaches I have digest--the human element.
I hear a lot of discussion about technical approaches to unwinding the bits and bytes of technical debt, but very little guidance for anyone on how to unwind the human aspect of all of this, acknowledging the years of business and political decisions that contributed to the technical debt. Its one thing to start breaking apart your databases, and systems, but its another thing to start decoupling how leadership invests in technology, the purchasing decisions that have been made, and the politics that surrounds these existing legacy systems.
I don't know about you, but every legacy system I've ever come across almost always has had a gatekeeper, an individual, or group of individuals who will fight you to the death to defend their budget, and what they know about tech (or do not know). I've encountered systems who have a dedicated budget, which only exists because the system is legacy, and with that gone, the money goes away too--sell me a microservices solution for this scenario!
Another dimension to this discussion, is that investors in microservice solutions are not interested in their money being used for this area. It just isn't sexy to be spending money on dealing with corporate politics, and unwinding years of piss poor business decisions, and educating and training the average business user around Internet technology. If you do not unwind these very human led, politically charged, business environments, you will never unwind the systems that exist within their tractor beams. Never. I'd care how much YOU believe.
In the end, I'm not trying to make you feel like you are going to fail. My goal is to encourage more investment in this area by the microservice pundits, vendors providing solutions the space, and VC's who are pouring money into these solutions. Many of the young, energetic folks at the helm of startups do not fully grasp the human side of corporate operations, and the potential quagmire that exists on the ground in front of them.
I am hoping that a handful of service providers out there can lower the rhetoric around their services and tooling, so that expectations get set at more realistic levels. Otherwise the push-back against the first couple failed waves of microservice implementations will become impenetrable, and blow any chance of making it work.
We closed the call for papers for APIStrat Austin 2015 last week, and have been processing the almost 100 talks that were submitted. We already had some session topics identified for the API conference this November 18th, 19th, and 20th in Austin, TX, but were also delighted to see the vast number of topics submitted through the call for papers process, resulting in this tag cloud.
This edition of APIStrat doesn't have a theme--we are just making about the roots of the conference, and looking to focus the conversation around the most pressing issues we face across the space at the end of 2015--I think this tag cloud reflects that conversation.
We have gone through all the talks, grouped them by topic, and will be reviewing them with the session chairs we have assigned to help organize and manage each track. By October, we will begin publishing the calendar, and sharing more detail about each session track, and the speakers in attendance.
While call for papers is closed, we have a handful of sponsorship opportunities left, so contact us if you want to help support the conversation at APIStrat. At the very least, make sure you are registered before the very intimate event sells out, you won't want to miss the conversations that occur.
Yeah, I know, it is a click-bait title. My goal is to not flame the haters, but educate everyone involved, both the people looking to learn about microservices, and those who are practicing microservices, so I'm hoping you will forgive me--my goal is not page views.
I'm sticking to my goal of not making microservices a regular thing I talk about, but I wanted to share some observations after listening to folks talk at Gluecon last week--a week that was full of microservices love (and hate). In the past, I mentioned that I received a handful of emails, and DMS from folks letting me know I'm not doing microservices, and after Gluecon I think I have a beginning list of why I think people generally are telling me I'm doing it wrong.
- Tooling - People tend to identify with the tooling they use, and deeply associate the function with the tool. So if I don't use Puppet or Chef, I'm not orchestrating. If your not using Jenkins, your not doing continous delivery, and so on. This is dangerous for microservices storytelling, but also is something that will continue.
- Vendor - This is probably the most egregious reason I see people telling others they are doing microservices wrong, and is somewhat akin to the tooling item, but is really just rooted in selling you something. You will never do it right, unless you are their customer, so don't worry about it.
- Scope - You just aren't doing it at the scope someone else is doing it. Microservices is something large enterprises do, not small companies, or individuals. You have to be operating across many availability zones, many servers, etc. Horseshit. Microservices can be done at any scope, don't listen to them.
- Completeness - You are only doing some of the aspects of microservices. Maybe you don't need to orchestrate across many operational zones, and your traffic looks very different than some of the usual suspects. Microservices is very seldom a complete picture, it is more about doing what works for you, cherry picking the valuable items you can apply in your world, not any complete definition.
- Incomplete Info - Someone just read a single blog post you wrote, and hasn't been following the series, asked any questions, and just make assumptions about a single thing you wrote. While I feel you should make each post as complete as possible, the responsibility is on the reader to do their homework - Bwahahaha! yeah right, pundits will never do that.
So far these are the biggest reasons that I've extracted from people me telling me I am not doing it right. Ultimately it comes down to power and control, which I saw play out with SOA, REST, Linked Data, and other areas of "API". They either want to make you feel small, or not cool because you aren't using the tools they are using, or they just want you to buy their snake oil.
In the end, for me, microservices are just a new storytelling package. They are mostly marketing, as many of the skeptics like to point out, and the companies who are truly doing microservices, will care about sharing their real-world stories, and work overtime to help you understand--there are a handful of them out there, like Runscope and Iron.io.
For me, I will be focusing on the moving parts of microservices, which is why you'll see me talking about API design, deployment, management, testing, monitoring, security over just "microservices" umbrella. I could easily attack many of the pundits who challenge me, and say you aren't doing API management, evangelism, monetization, or other critical areas right, but why would I? Where does that get us?
As usual, if you have any questions, let me know how I can help. I'm happy to share, as I try to make sense of all of this. I'm adding some new research sites to support the conversation.
I've been monitoring an emerging slice of the API, that has been dubbed "microservices" for some time now, and you've even heard me explore its use, when describing my architectural approach to redesigning more core API stack for my internal systems. I’m slowly redesigning my internal API stack, keeping them as small as possible, throughout all aspects of operations, deploying each endpoint in a single dockerized container, that contains the OS, database, web server and all server side API code. I'm also using Github, in conjunction with APIs.json to assist me in my orchestration, throughout every aspect of these APIs lifecycle from design to testing—all a new approach for me when managing my APIs.
Is defining my APIs in this way, so that the definition is much more than just the actual API endpoints, but also include the entire backend, and lifecycle workflow Microservices? Fuck if I know. One thing I do know, is I've had enough conversations with folks who are doing microservices, resulting in me saying several times that the micro services definition is something very personal. Something which the first 10 times I said , sounded very positive, but after experiencing multiple people telling me what I'm doing isn't microservices, I can tell the term definitely will continue to be very personal—very much in the same way conversations around REST or Hypermedia has gotten personal on forums, and HN threads.
You know what this tells me about microservices? Is it is more about power, control and influence, everything from enterprise architects stating their endorsement of the concept, down to the individual architects, who love to make sure you know that the way you are doing it, is WRONG! I don't think there is a single definition of "microservice", and much like REST, I think we'll hear plenty of “right” definitions, by the enterprise justifying what they are doing, and those who are selling something to the enterprise. Is there anything wrong with this? Nope. Just not my style, not my game, and has shown me over the couple months how the concept is not a fit for me.
I’m going to stick in the realm of API. It is a completely bullshit term, that means everything, but you know what? Somehow it escaped the IT, architect, and vendor ownership that SOA and now microservices possesses. For me APIs have long been more than just the tech, it is also about the business and politics of any platform implementation, so why can’t it contain my architecture styles as well? So from here forward I will be closely paying attention to microservices conversations, but you won’t hear me use the word, as I’m more comfortable in API land, and what it has come to mean to the wider public—I just don’t give a shit about what the enterprise adopts, or doesn't.
Microservices will reflect the power and control of the entities behind them, unlike APIs, which also possesses that characteristic, but also quickly shifts the conversation to be more about the innovation and opportunity that developers bring to the table, and the end-users who put that to use. Let’s make sure as we do not get caught up in the architectural discussions behind the technical curtain, and that we remember why all of this API stuff is working. Let's not get caught up in all the value being just about our architectural styles, and re-live a classic IT tale, ignoring that the real value is what people actually do with our services.
The Enterprise Will Make The Same Mistakes With API And Microservices That They Did With SOA, Because Essential API Concepts Go Right Over Their Head13 Mar 2015
I would say the enterprise space fleet has successfully shifted their course, heading in the general direction of everything API. SAP, Oracle, IBM, Microsoft, and the rest of the usual suspects have pledged their faith to APIs, mobile, and IoT, all in a very public way. For those of us who jumped off the SOA bandwagon a while back (getting on the API bandwagon, yeehaw), this can be amusing to watch, cause the API bandwagon is better y'know?. ;-)
I sincerely hope that these large companies can make a shift towards an API way of life, or a microservice devops way of operating, or whatever the cool enterprise kids are calling it these days. Some companies will find success in these areas, while many, many others will just talk about it, and buy a wide range of vendor services that will help them talk about it—there will be a lot of money to be made (and spent).
The thing I fear, is that many of the core principles of why this whole thing is working will go right over the heads of many technologies and business leaders in the enterprise. Elements like simplicity, transparency, openness, and functioning in a decoupled, decentralized way. You are the enterprise, many of the essential ingredients of API, are just fundamentally at odds with what you are. You are a big, complex, powerful, and political corporate presence—a beast that does not like being taken apart, reinvented over and over, and being forced to work in concert with other internal, partner, and public resources, in an open way.
This doesn't mean that some enterprise entities won’t be successful—some will. I will be carefully watching the waves of enterprise organizations, for these tech savvy few, who internalize the core philosophy around APIs, successfully uploading their corporate soul to the API singularity. However, I predict I will have to work very hard to find these companies amidst the wreckage of the enterprise organizations that make the same mistakes they did with SOA, because APIs are just not compatible with their DNA, and ultimately will be rejected for reasons that are more enterprise than they are API.
I'm just getting started exploring the ways to use APIs.json when it comes organize my new Docker fueled, micro services stack. I’ve been using APIs.son to describe each micro service, as well as define the overall collection of almost 20 micro services. I’m using the include collection, as a navigation for the loosely coupled stack of micro-services, and my friend Chris Spiliotopoulos (@chefarchitect), has done some more work with his Swagger.ed, delivering some APIs.json goodness that is in alignment with where I want to take all of this.
Adding to the discovery and visualization work he did for Swagger, Chris enabled Swagger.ed to look for valid APIs.son files as well, so when you browse to any APIs.son, like the one I have for my micro services stack, you see the little APIs.son search icon in the address bar.
When you click on the APIs.son search icon in the address bar, you get a very cool visualization. Its pretty basic at the moment, just a visual catalog of the APIs available in the include collection of my stack, but when you connect up with the Swagger visualization work he's already done, we could have a pretty cool API catalog for managing and exploring microservices.
I have almost 20 micro-services listed, and Swagger.ed gives me the ability to navigate it in a very interactive way. Whats next? We don’t know…it is about exploration, and finding out the most meaningful way of exploring the APIs I deploy and aggregate into APIs.son collections.
I can see having a visual catalog of all of my API design that I collect, then deploy, and evolve them as needed for various parts of my infrastructure or for my clients, into other sub-collections? Disconnected collections? Loosely coupled collections? Not sure how tight I want things, part of the micro service definition for me is the size of the network connecting the services, as well as the services theselves.
API Management Infrastructure And Service Composition Is Key To Orchestration With Microservices In A Containerized World02 Feb 2015
As I work to redefine my world using microservices, I have this sudden realization how important my API management infrastructure is to all of this. Each one of my microservices are little APIs that do one thing, and do it well, relying on my API management infrastructure to know who should be accessing, and exactly how much of the resource they should have access to.
My note API shouldn’t have to know anything about my users, it is just trained to ask my API management infrastructure, if each user has proper credentials to accessing the resource, and what the service composition will allow them to do with it (aka read, write, how much, etc.) My note API does what it does best, store notes, and relies on my API management layer to do what it does best--manage access to the microservice.
This approach to API management has llowed me to deploy any number of microservices, using my API management infrastructure to compose my various service packages—this is called service composition. I employ 3Scale infrastructure for all my API / microservice management, which I use to define different service tiers like retail, wholesale, internal, and other service specific groupings. When users sign up for API access, I add them to one of the service tiers, and my API service composition layer handles the rest.
Modern API management service composition is the magic hand-waiving in my microservice orchestration, and without it, it would be much more work for me to compose using microservices in this containerized API world that is unfolding.
Disclosure: 3Scale is an API Evangelist partner.
I'm using APIs.json to organize my swagger defined microservices running in docker containers, and using the machine readable API index to drive navigation between microservices organized in a single collection. APIs.json provides a simple, machine readable way to index the technology, business, and political elements of each microservice I deploy.
As I was auditing the 18+ microservices I’ve setup for my core operations, I wanted to audit each one, and make sure I had included various elements in each APIs definition, like where developers can onboard, terms of service, and where to find related code samples. To accomplish this, I was able to just compare each microservice APIs.json, with a master APIs.json template I had established while planning my original microservice stack.
Sometimes it is just as important to know what is missing, right alongside what is available for all of my microservices, and APIs.json is proving to be a simple way of understanding this layer of my API operation at scale.
If you think there is a link I should have listed here feel free to tweet it at me, or submit as a Github issue. Even though I do this full time, I'm still a one person show, and I miss quite a bit, and depend on my network to help me know what is going on.