Archive for the ‘Kiln’ Category

Fog Creek and Heartbleed

April 10th, 2014 by Mendy Berkowitz

Along with the rest of the internet, Fog Creek has been reacting to the Heartbleed vulnerability which was discovered this Monday.

TL;DR: FogBugz and Kiln On Demand, Copilot and the Fog Creek website were not vulnerable. Trello was vulnerable and has been remediated.

Fog Creek handles SSL connections at the load balancers in front of the application servers. The load balancers for FogBugz and Kiln On Demand, Copilot and the fogcreek.com website are using the 0.9.8 branch of OpenSSL. This version does not have the newer heartbeat extension and is therefore not vulnerable to Heartbleed. None of your data on fogbugz.com, kilnhg.com, copilot.com or fogcreek.com has been exposed to the Heartbleed vulnerability.

The load balancers for Trello were using the 1.0.1 branch of OpenSSL and were vulnerable. We have upgraded OpenSSL and replaced the Trello certificates. More details and important recommended steps for all users are available in a post on the Trello blog.

We have also reviewed our vendors (for things like this blog) to ensure that our other SSL certificates were not potentially compromised. Fortunately we have not had to replace any other certificates.

If you have configured web hooks in your Kiln On Demand account, we recommend that you verify that the target servers have not been affected. If you use one of the pre-configured types, or your custom web hook uses HTTPS, the data sent to that external service may be at risk. We do not however, send any of your Kiln login credentials, so they are safe.

We take the security of your data seriously. We are committed to protecting it and to maintaining clear and open communication with you. If you have any questions, comments or concerns please contact us.

The Kiln Bottleneck

April 3rd, 2014 by Jacob Krall

Early in February, one of our Kiln On Demand customers sent us an email to let us know that Git clones were taking much longer than before. Our stellar support engineers started measuring Git transfer times from their own personal accounts, and confirmed that Kiln clones were being sluggish. We didn’t know what the problem was, and since it wasn’t completely preventing the clones from succeeding, we prioritized further investigation lower than we should have.

On Friday, March 7, another customer reported the same issue and wanted to know if we were throttling their connection. Our existing monitoring showed all services performing nominally, especially after the Kiln SSH server resource utilization fix in Kiln 3.0.115 had been released, so we were still a bit puzzled. I started writing a little PowerShell script I called timeit.ps1 to measure the time it took to run git clone and hg clone against the various distributed services inside Kiln. (Unlike Trello’s codenames, mine are boring and vaguely descriptive.) I ran timeit.ps1 against my local machine, and saw nothing unusual.

Kiln DVCS Hosting Infrastructure

Here are the endpoints that timeit.ps1 attempts to clone from; once with Git and once with Mercurial.

HAProxy, IIS, and Apache on the web side; HAProxy, Tesseract and KilnSSH on the SSH side.

HAProxy terminates HTTPS traffic and sends the request on to an IIS server.

IIS authenticates the git/hg request and forwards it to Apache on the backend server that holds the requested repository.

Apache hosts the process that actually performs the git/hg transfer.

HAProxy forwards all SSH packets, untouched, to a Tesseract server.

Tesseract reads the SSH traffic and forwards it to KilnSSH on the backend server that holds the requested repository.

KilnSSH authenticates and performs the git/hg transfer.

Interesting Problems Happen At Scale

There are very few differences between my local development install of Kiln and the live Kiln On Demand service. One particularly important difference is that Kiln On Demand serves real customers, while my laptop is dedicated to just one user (me). Since I couldn’t reproduce the problem on my local machine, but our support engineers and customers could see it every day, I knew the problem had to be something environmental. Monday morning, I tweaked timeit.ps1 to run against any Kiln instance, and pointed it at my personal Kiln On Demand account. By the end of the day on Monday, I shared this compelling graph with the rest of the team in chat.

The public endpoints are much slower than the endpoints immediately behind them

Bingo. Our load balancer, HAProxy, was doubling the clone speed. Kiln QA engineer Andre confirmed from his home in Vancouver, B.C. by running timeit.ps1, he got the same exact shape on his graph. It had to be the load balancer!

By Tuesday morning, our System Administrators (whom I had otherwise forgotten about) had already formulated a plan to add power to the HAProxy machine. A Puppet script was written and run against the staging environment for testing on Tuesday. The process was repeated on our internal Kiln instance Wednesday morning for dogfooding and internal testing all day. That Wednesday night’s maintenance successfully switched the customer-facing servers to the new configuration, and the results were immediate and drastic:

after

Mercurial KilnSSH clones are somehow slower than its caller, Tesseract. This is a strange artifact of the measurement.

Richard, our support engineer in London, had a monitoring task set up that ran a git clone via public SSH. His graph confirmed that the HAProxy maintenance had an immediate and drastic effect. The variance dropped dramatically, making a smooth, placid lake out of what was once a spiky mountain range:

A graph showing Git clone becoming faster and less variant.

Lessons Learned

The scientific method is insanely effective. Formulate a hypothesis, design an experiment, run the experiment, and analyze the results. Then it is often obvious what you need to do.

Software is invisible. You can only observe its behavior indirectly, by measuring it. That means you can’t see what you don’t measure.

Your customers know when your application has a performance problem. If you can’t explain it, you need to dig in and figure it out.

Next Steps

timeit.ps1 was a one-off test script. It only runs in Windows, and is not very easy to use. We are adding automated measurement of DVCS metrics to our monitoring systems.

Start a free trial of Kiln and enjoy the faster Git and Mercurial clones today!

What’s new in Kiln? AppVeyor web hook!

October 17th, 2013 by Kevin Gessner

Do you use AppVeyor to build and deploy your .NET projects?  If so, you’re in luck: Kiln’s web hooks now connect to AppVeyor!

AppVeyor web hook

AppVeyor can pull your code directly from Kiln.  Once you’ve created your project in AppVeyor, you can trigger a build manually, or set up the new web hook to trigger a build on every push.

AppVeyor builds

Building .NET projects has never been easier!

The AppVeyor web hook is available in Kiln 3.0.90 and higher.  Want to see Kiln integrate with another service?  Let us know!

Introducing name association in Kiln

October 2nd, 2013 by Hao Lian

My first Hotmail account was “hao2lian@hotmail.com” because at the time “hao” and “haolian” were taken and I didn’t know any better than to add an ungainly two in the middle of my name.

Then when I discovered IRC my mom, bless her heart, told me I could have any handle I wanted as long as it didn’t contain my real name. (I was 14 at the time, and Edward Snowden was 21.) I wanted to keep being hao2lian, but I also wanted my mom to be happy. So I conceded.

Now I had two identities. And I was hooked, hooked on coming up with better and better nicknames for myself until I had to start tracking all my usernames in a text file, which by the end of the aughts included multiple Yahoo and Gmail email accounts. Thank goodness for Dropbox when it finally came along.

So it’s not surprising that name and email association is one of the hotly requested features for Kiln. We took a while, but we finally got around to it. As part of his last summer’s Kiln internship, Josh Cooper spruced up the changeset page with this nifty link, which you’ll see if you’re an administrator on your Kiln account:

The name association popup

Were you to click on the link and choose a Kiln user

A successful name association

you will have taught Kiln that the username maps to that Kiln user! From that day forward, all commits with that username will become linked to that person in your Kiln account. Kiln beams at you. Armed with this new information, it can connect up your search results, revset filtering, review data, and more to the right Kiln account. All for you, and your ever-mercurial internet identity.

Hao Lian is a programmer on the Kiln team. Did you know about the ArtisanVideos subreddit?

Introducing the Kiln Command Line Tool for Git

September 25th, 2013 by Benjamin Pollack

We made Kiln into the best enterprise Git hosting tool in March, and we’ve been happily shipping Git and related utilities as part of the Kiln Client Tools since then. But there was always one component missing: an equivalent of the hg kiln command-line tool.

We know that many users prefer to use GUIs, but for those of us who are wedded to our command lines, having a quick and easy way to work with Kiln without leaving the keyboard is a godsend. From Mercurial, this is pretty easy: want to see a list of related repositories? That’s just hg kiln -t. Want to make a new repository to push your changes to? A quick hg kiln -n MyBranch and you’re good. Want to see a given file annotated with Kiln’s multilevel line annotations? hg kiln -a FooBar/Baz.cs and your browser pops up before you have a chance to spill your coffee. But since hg kiln was implemented as a Mercurial extension, Git users were left out in the cold, with no equivalent functionality.

No more. Now, Git users now have the git kiln command to access equivalent functionality. Want to see related repositories? That’s a simple git kiln related. You can view commits in Kiln via git kiln show (which takes everything Git understands, including even weird things like HEAD^^), where you can open reviews, link bugs, or view them in the Electric DAG. You can create branches in Kiln via the simple git kiln create-branch command. You can even do something your Mercurial colleagues can’t yet do: look up the Mercurial changeset that corresponds to the a Git commit, with the handy git kiln hg-sha command. There’s a lot more built-in, too; you can get the full details via git kiln help.

While we’re only distributing this first release of git kiln with our Windows Kiln Client tools, git kiln already runs on Mac and Linux. We’ll be working in the coming weeks to provide an easy way for Mac and Linux users to install it. In the meantime, git kiln is fully open-source and easy to build with Go, so feel free to take a look, play with it, and even submit a pull request if you’re so motivated.

What’s New in Kiln? The Intern Edition

August 12th, 2013 by Kevin Gessner

Kiln’s summer interns have been hard at work adding a bunch of great new features. Here’s a few of the cool things they’ve done so far.  All of these features are available in Kiln On Demand right now — sign up for a free trial to try them out!

Adding & Removing Bookmarks & Branches

Kiln has long supported moving Mercurial bookmarks and Git branches, but creating or deleting bookmarks required the command line. Now, just like tags, you can create bookmarks and branches directly in Kiln:

blog_add (1)

Once a changeset has been tagged with a bookmark or branch, it can be moved like any other bookmark on the main repository page. Have an extra bookmark or branch?  They can now be deleted from both the individual changeset pages as well as the repository DAG views.

blog_delete

Choose Your Own Encoding

File encodings are a fact of life for modern programmers. In a perfect world, everyone would use the same encoding. However, we don’t live in that perfect world, and in practice, your repository might contain text files in numerous encodings. In the past, Kiln didn’t recognize that encodings other than UTF-8 even existed, and would show a less-than-helpful “This is a binary file” message for text files in encodings like UTF-16, Shift JIS, and Latin-1.

That’s all changed! When viewing a file, you can now select which encoding you want Kiln to use to render the text.  (If there are errors in that encoding, you can still see the results, errors and all — just click the “Decode” button.)

Don’t see the encoding you use? Contact Fog Creek Support and we’ll see if we can add it!

Draggable Branch Repositories

To help your team collaborate, Kiln lets you mark your repositories as “central” and “branch” repositories, distinguishing your important trunk or release code from dev’s workspaces. You’ve always been able to drag and drop central repositories to rearrange the project page, and the branches would follow along.  You could promote a branch repository into a central one, but couldn’t demote a central repository to be a branch. So we said, “Hey, why don’t we let people do all of those at once?”

That’s right, now you can go onto any project’s page and drag-and-drop away. Reorder central repositories, reorder branch repositories, make branch repositories central, and make central repositories branches. You can do it all with a simple click and drag of the mouse. Feel free to reorganize your project as you see fit!

Want to work on awesome projects like these? We’ll begin accepting applications for our Summer 2014 internship next month!

What’s New in Kiln Harmony? Spiffy Project Management!

July 29th, 2013 by Hao Lian

The project page has been a mainstay of Kiln ever since its first release. It’s been the place where at a quick glance you can view and manage all the repositories in your project. And it’s been a beacon for Kiln, symbolizing our manifesto to enhance collaboration and organization among programmers. When we planned all the new features for Kiln Harmony—seamless translation between Mercurial and Git version control systems, parsing in F#, SSH support, and more—we went home and slept soundly, knowing we had the razzle-dazzle down. But the project management user interface, long neglected, had rusted over the years. It nagged at us; a shadow in our dream world. So we woke up and went to the elbow grease store. Then we went to the spit and polish store. Then we went to a restaurant, because we were hungry. And now here are some of the changes we made to get to this final result:

The New Kiln Project Management User Interface

Permissions

The permissions popup.
A drag-and-drop permissions system shows at one glance what permissions each user has. The raw power of FogBugz and Kiln’s permissions system, made accessible.

Context menu

The context popup.
A revamped context menu highlights all the tasks you can do straight from the project page: branch, clone, and archive a repository.

Conflict detection

The reload notification.
The project page can reload itself when someone else makes a change, preventing conflicts and strengthening friendships. Essentially the project page is now a modern JavaScript one-page application written with Knockout. The old project page was a loosely-linked collection of popups and web pages, and one of our goals was to give everything a unified look and feel.

Drag and drop

A repo being dragged and dropped.
Reordering repositories is as simple as a drag and drop thanks to jQuery UI.

Click to edit

Changing a project description.
Rename your project or edit its description in one click.

Aliases

The aliases list.
Aliases can breathe freely now, liberated from their repository’s box. Add as many aliases as you want; they’ll all show up.

With all these features, you could even say that Kiln Harmony makes project management fun! But you shouldn’t, because you sound like a robot when you do.

Picture of a cat robot.
A cat robot.

Hao Lian is a software developer for Kiln. Did you know about the Shift-Space keyboard shortcut?

What’s new in Kiln On Demand? View a diff between any two changesets

May 17th, 2013 by Kevin Gessner

One of the first features we added to Kiln was diff viewing: click a changeset, see what was changed. In terms of raw page views, viewing diffs is one of the most popular features in Kiln.

But viewing a single diff isn’t always everything you want. Often, you’ll want to see all the changes made between two changesets that are farther apart—perhaps the difference between two tagged releases, or the sum of the changes made in a feature branch. With Kiln Harmony, now you can!

Simply load any changeset, and you’ll see the diff from the changeset’s first parent. Click the “Diff from another changeset…” link to search for another changeset (you can search by commit hash, tag, and branch names, or phrases in the commit message, or even filter by author and date—it’s the full power of Kiln’s search engine). Click the results to view the diff from that changeset!

Changeset search

You can see some examples on our demo site:

This feature is available in Kiln 3.0.33 and higher. Sign up for a free trial and try it out!

Dive into parser combinators: parsing search queries with F# and FParsec in Kiln

April 23rd, 2013 by Hao Lian

We open on: the past

The year: 2012. The problem: search. With a new release of Kiln, search is now forefront and center. You can zip around repositories or code with a simple tap of the keys, and boy is the future bright.

Powering search was our search engine. And powering it was our search-query parser, a couple hundred lines of code that parsed a query into a list of keywords and filters. For example, if you asked of Kiln

foo bar project:Eggs date:yesterday..now author:Tyler

Kiln finds all the commits, by people named “Tyler,” to a repository in projects named “Eggs” since yesterday with the words “foo” or “bar” in the commit message.

But try to search "foo bar" and you would be disappointed. The unspoken rule of the internet is that surrounding two words in quotation marks should make a search engine look for both words as one phrase instead of two separate words. So "foo bar" should match the string “boy I had a lot of foo bar pie” but not the string “foo and bar are two friends from way back when.” Pretty goofy rule, but the internet is a goofy place.

It’s 2012, and Kiln does not have phrase search. We left it on the cutting floor to make room for everything else we wanted, and we regret it. Life moves on.

And we cut to: the present

The year: 2013. Not having phrase search: more and more irritating. Having migrated our full-text indexing to elasticsearch, phrase searches are not only possible but easy. So you, being a developer on the Kiln team, don glasses and open the .cs file containing the query parser. Written in C# and presented for your consideration is a jumble of grammar rules and intermediate parse trees, a jungle of loops and state. A flock of crows take off from a nearby tree. You close the file.

“This seems like the ideal intern project,” you think to yourself. “It would be a shame to not allow someone else to rewrite this.”

Just then, Andrew Pritchard walks by your office. Andrew Pritchard was our summer 2012 intern who worked on a dazzling array of Kiln features, including phrase search. We will borrow a hypothetical version of him. Look at him, walking with the smooth confidence of a man not yet burdened by string parsing.

“Help us, Hypothetical Andrew Pritchard,” we said. “What do you know about parsing?”

H.A.P. points you at FParsec, a parser combinator library for F#. He begins erasing your whiteboard and drawing diagrams while you wonder what he is talking about.

“Hold on,” you say, slapping the multi-colored markers out of his hand. “I have many reservations about what’s happening right now but here’s the biggest one. F# is that that new functional-programming language from Microsoft right? Kiln is a giant ASP.NET MVC application that uses C#. There is no room for F#, Hypothetical Andrew Pritchard, you crazy lovable human being you.”

“No,” he replies. You two stare at each other for a while.

It turns out that .NET’s Common Language Runtime, plus increasingly better F# support in Visual Studio, lets you create an F# library inside your solution and reference it from a C# project. There are some quirks: ReSharper support for F# is ongoing, F# files have to be sorted in the solution tree in the order you want them to be compiled, and F# collection types map awkwardly from and to C#—to name three big ones. Overall though the experience is surprisingly pleasant. I say in the year 2013 you can (and should) alternate between F# and C# depending on the problems you are solving.

We created an F# project with the source code in this blog post if you would like to follow along. If you are not familiar with F#, fear not! By and large the F# syntax can be intuited; for a look-see, Wikipedia also has a buffet of code snippets. On my part I’ll use highly descriptive variable names and mention C# analogues to F# features when possible.

“FParsec is great, but we need F#. No biggie,” H.A.P says, shrugging his shoulders. “Besides, F# is functional, which means it’s ideal for a self-contained, computer-science-y project like string parsing.”

“It is fun.”

“You will like it.”

You are sort of convinced. In any case, he has covered your whiteboard in figures and symbols. He looks at you, then looks at the board. He walks over and gently pushes you out of your chair. You get up, brush yourself off, and read the notes on the whiteboard as he begins typing into your computer. Which notes are:

(more…)

What’s new in Kiln? Sprint.ly integration!

April 3rd, 2013 by Kevin Gessner

Do you use Sprint.ly for project management? Kiln now has a web hook that you’ll like! You can manage your Sprint.ly items using special comments in your commit messages.

sprint.ly integration

Simply enable the web hook in your Kiln account, then reference Sprint.ly items using Sprint.ly’s commands. As you push to Kiln, you’ll see your changesets linked from your Sprint.ly items.

This web hook is available in Kiln 3.0.28 and higher. Sign up for a free trial and try it out!