Our paper ‘PelVis: Atlas-based Surgical Planning for Oncological Pelvic Surgery’ was accepted for presentation at our largest conference, VIS (and publication in IEEE Transactions on Visualization and Computer Graphics)!
Abstract: Due to the intricate relationship between the pelvic organs and vital structures, such as vessels and nerves, pelvic anatomy is often considered to be complex to comprehend. In oncological pelvic surgery, a trade-off has to be made between complete tumor resection and preserving function by preventing damage to the nerves. Damage to the autonomic nerves causes undesirable post-operative side-effects such as fecal and urinal incontinence, as well as sexual dysfunction in up to 80 percent of the cases. Since these autonomic nerves are not visible in pre-operative MRI scans or during surgery, avoiding nerve damage during such a surgical procedure becomes challenging.
In this work, we present visualization methods to represent context, target, and risk structures for surgical planning. We employ distance-based and occlusion management techniques in an atlas-based surgical planning tool for oncological pelvic surgery. Patient-specific pre-operative MRI scans are registered to an atlas model that includes nerve information. Through several interactive linked views, the spatial relationships and distances between the organs, tumor and risk zones are visualized to improve understanding, while avoiding occlusion. In this way, the surgeon can examine surgically relevant structures and plan the procedure before going into the operating theater, thus raising awareness of the autonomic nerve zone regions and potentially reducing post-operative complications. Furthermore, we present the results of a domain expert evaluation with surgical oncologists that demonstrates the advantages of our approach.
I recently mentioned a certain OAH webviewer short paper that was accepted as a EuroGraphics Education paper, but I now have some more news to share: the webviewer will be used in a Coursera course on anatomy of the abdomen and pelvis that starts this week, entitled ‘Anatomy of the Abdomen and Pelvis; a journey from basis to clinic.’! This course is a Massive Open Online Course (MOOC), a free online course aimed at unlimited participation, and organized by highly qualified people at the Leiden University Medical Center.
So far, already 7,796 people signed up. If you also have an interest in anatomy, why not sign up yourself? It’s completely free and completely awesome, I promise! I do have to add, it is not for the faint of heart, since it features live anatomical dissection videos. I am following the course myself and really enjoying the classes. I even just passed my first quiz with 100% correct answers. Try and beat that 😉
Soon, you could be playing around with this yourself:
Looking forward to seeing you as my virtual classmates in the course 🙂
So until last year, I never worked with shaders in VTK (the Visualization Toolkit). This was kind of sad actually, but I thought it would be super difficult, the need didn’t really arise and well, ain’t nobody got time for that! For our VIS 2016 submission though, I finally got to play around with them. And spoiler alert: it’s not that difficult and even kind of fun! Especially with a little help from highly intelligent co-authors 🙂
So today I would like to share a little Python example I made as a tutorial, featuring a cel-shaded (or toon-shaded if you’re one of those people) donut or skull in exactly 100 lines of code, probably half of them comments. Before we get started, a little sneak preview:
Right, so let’s get started! I’m using VTK version 6.3 built with the ‘old’ OpenGL backend and not the newer OpenGL2 one, in which custom shaders are handled in a different way. I’m using 64-bit Python 2.7 to go with this. You can find the skull surface and code you need here: shaderfun.
VTK supports GLSL shaders that you can add to your vtkActors. The easiest way to do this is, in my opinion, by specifying your shaders as multi-line strings using triple quote block (”’ bla ”’ or “”” bla “””). You can define a vertex and fragment shader in this way. To get your shader to work with VTK, you need to define a propFuncVS in the vertex shader, and a propFuncFS function in the fragment shader. These are the ones we’re using today:
The vertex shader:
vert = """
varying vec3 n;
varying vec3 l;
n = normalize(gl_Normal);
l = vec3(gl_ModelViewMatrix * vec4(n,0));
gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
Once you made some shaders, you want to add them to your vtkActor so that VTK knows which objects to shade. First you make a vtkShaderProgram2 and set its context to your render window. Then you make a vtkShader2 for every shader you made, e.g., one for the vertex shader and one for the fragment shader, set the source codes to your multiline GLSL strings. and set their context to that of the vtkShaderProgram2. Now you can add the shaders to the program. You get the OpenGL property of your actor and call SetPropProgram to attach your shader program to the actor. Finally, just set the Shading to On of your property and you are good to go! Did I make it sound difficult? It’s not, here’s the relevant code:
# Now let's get down to shader-business...
# First we make a ShaderProgram2 and set it up on a date with the RenderWindow
pgm = vtk.vtkShaderProgram2()
# For both the vertex and fragment shader, we need to make a Shader2
# Also set them up with the RenderWindow, by asking the ShaderProgram2 for an introduction
shaderv = vtk.vtkShader2()
shaderf = vtk.vtkShader2()
# Now we add the shaders to the program
# And tell the actor property that it should totally use this cool program
openGLproperty = actor.GetProperty()
That’s it for my basic tutorial on how to use shaders in VTK. If people are interested, I could make a more elaborate tutorial on how to pass attributes, use textures, and scalar mesh properties in your VTK shaders 🙂 Hope this was helpful!
The thrice rejected, thrice cursed work on visualizing anatomical variations in branching structures has been accepted as a short paper at EuroVis 2016! This means I can finally show you a video of the work without jeopardizing the double-blind review process:
Abstract: Anatomical variations are naturally-occurring deviations from typical human anatomy. While these variations are considered normal and non-pathological, they are still of interest in clinical practice for medical specialists such as radiologists and transplantation surgeons. The complex variations in branching structures, for instance in arteries or nerves, are currently visualized side-by-side in illustrations or expressed using plain text in medical publications.
In this work, we present a novel way of visualizing anatomical variations in complex branching structures for educational purposes: VarVis. VarVis consists of several linked views that reveal global and local similarities and differences in the variations. We propose a novel graph representation to provide an overview of the topological changes. Our solution involves a topological similarity measure, which allows the user to select variations at a global level based on their degree of similarity. After a selection is made, local topological differences can be interactively explored using illustrations and topology graphs. We also incorporate additional information regarding the probability of the various cases. Our solution has several advantages over traditional approaches, which we demonstrate in an evaluation.
Medical visualization isn’t always about rendering data in the most realistic way possible. In fact, quite often it isn’t. Illustrative rendering techniques have been developed that present medical data to the viewer in a style that is more abstract, often emphasizing important features, while taking emphasis away from the less relevant. Illustrative rendering techniques can even make data originating from medical imaging scanners such as CT, look hand-drawn. Check out for instance the works by Tobias Isenberg, Stefan Bruckner, Ivan Viola and Kai Lawonn to name a few. As you may recall, last year’s VCBM paper I had the pleasure of co-authoring also involved illustrative rendering:
So what you see here kind of looks like a sketch hinting at the shape of the human body on a table right (well with some limbs missing ;))? Actually it’s a rendering of a CT scan combining toon shading and feature lines without any artists involved. Here’s another example by Roy van Pelt:
He visualized blood flow using particles that get elongated as they move faster in combination with speed lines indicating the direction and speed of the particles. This style is reminiscent of something you could see in comic books or cartoons. In fact, I think there is a lot we can learn from comic book artists that we can apply to medical visualization.
For this reason, I asked Gerrit Rijken, AKA Iosua, a freelance illustrator as well as comic book aficionado, to write about the basics of inking, and more specifically the do’s and don’t’s and why’s of it all. In his elaborate post, which you can find here, he talks us through many things that are of interest in illustrative rendering as well. He describes how inking techniques are applied to create textures, the importance of line weights, spotting blacks, screen tones, feathering and cross-hatching. He provides explanations of these techniques combined with examples illustrating the concepts.
I see many parallels to visualizing techniques. For instance, focus-and-context techniques and depth cues are relevant for both comic books and medical visualizations. There are also some interesting rules on line thickness that I had not considered before. Furthermore I see techniques that have already been adopted in medical illustrative rendering, such as stippling and hatching, while I did not see researchers using feathering yet (correct me if I’m wrong ^^). I hope you find this as interesting as I did and if you have further questions, do not hesitate to contact him!
So I guess we all know rainbows are bad, mmmkay. That’s why I wrote a lengthy blogpost for medvis.org and more recently a follow-up featuring four new pretty colormaps designed for Matplotlib. I realized though that I did not see the prettiness applied to medical datasets so I went to town with Paraview 5.0 (which includes the new colormaps by default now). I took these colormaps for a spin on a slice of the CALIX dataset from the OsiriX dataset collection, an arterial phase CT scan of the abdomen (and part of the thorax as you see below). I wanted to put the four new colormaps (Magma, Inferno, Plasma, and Viridis) side-by-side with the traditional grayscale and Jet (AKA rainbow) colormap to show what they look like on a medical dataset:
Looks kind of artistic, am I right? I would totally buy a print and frame it! So the gray scale is doing fine of course, and what we are accustomed to in medical images. The banding in the Jet colormap focuses the attention on the vertebra and ribs, which is probably not what you are actually interested in. Also information is lost in the soft tissue areas (muscle and fat). The four proposed colormaps are doing fine, though Plasma and Viridis start from blue tones, which is probably not what you expect when viewing on a black background, but can work well in case the low intensity regions contain information you actually want to show on a black background. I would love to compare with Matlab’s Pareto, but hey, proprietary, what’s a girl to do ^^
Now, of course, there is not much reason to replace the gray map when viewing single modality medical imaging data, but it could be interesting to consider for multimodal fusion viewing. For instance, when PET and CT are combined (either from a hybrid scanner or by software registration), often the anatomical CT data is represented in a gray colormap, while the functional metabolic information from the PET is shown overlaid in a color colormap (lol). This works well because both colormaps can be perceived simultaneously very well. For this, often a heated-body colormap is used, but you could also consider these four new options, as this is not a standardized choice and varies per manufacturer.
So far I’m talking about 2D representations, such as the traditional slice views above. When rendering 3D scenes though, and for instance wanting to map a scalar value on a surface to a color, one should be careful with these new maps. Since they are not iso-luminant, the changes in luminance may interfere with surface shading from lighting. Let’s take a look at a silly example:
This is a bit of a stupid example, because I don’t have proper data I can show you (I am only working with the most top-secret rectum information). But what we see here is a surface model of the os coxae with the normal Z component mapped to the Magma colormap. So what’s shading and what’s normal information in this case? Who’s to say? I am obviously:
So this is why you need iso-luminant colormaps for 3D surface information mapping, kids! Also, consider the Plasma, Inferno, Magma and Viridis for your 2D visualizations because:
They are beautiful
They are colorblind safe
They are printer-friendly (this means printers become happy when you print them (j/k, it means when you print grayscale it still works))
So last, well, let’s go with week, I hinted at a notification for a submitted EuroGraphics Education paper. I got said notification last week already, but was too busy with the EuroVis (EuroEverything!) Short Paper deadline to post about it.
As there is some sort of spoiler already in the title, it is probably not so surprising anymore, but there is an unexpected twist ;)… Our EuroGraphics 2016 Education paper on ‘The Online Anatomical Human: Web-based Anatomy Education’ was accepted, as a, drumroll please, short paper! Since I’m in my shortening-8-page-papers-to-half-their-size-mode (see also above: EuroVis Short Paper) anyway, good timing I guess 🙂
First up: a little In Memoriam for The ‘Weekly Status Update’. Remember those? I vaguely recall wanting to post weekly updates with five bullet points each. Ain’t nobody got time for that! I’m following up on excellent advice by cpbotha: “when under-achieving, lower your standards”. I’m no longer setting any expectations here, I just post whenever I want, whatever I want from now on. So what’s cooking?
Last week I briefly visited the University of Koblenz · Landau, in, you guessed it, Koblenz to work on a EuroVis STAR paper (time to shine) with the newly minted J.- PROF. DR. KAI LAWONN, who you may remember from:
If cpbotha can post after a four month hiatus, then so can I ^^. First of all, all the best for 2016 to all of you! I hope it’s a good one. The start of a new year is as good a time as any (if not better) for a look back, or review if you want to get fancy about it, of the previous year and to look forward to things happening in the time ahead.
First up, looking back:
I didn’t blog so much, because a) I was crazy busy (final year of the PhD anyone?), and b) personal issues that I will not discuss here. Maybe I will increase the update frequency, maybe I won’t. Wait and find out?
In September, I presented at VCBM! My favorite conference in the world (eat it, VIS!). It was in Chester, UK this year (full report here) and I presented work festively entitled “Illustrative Multi-volume Rendering for PET/CT Scans”, which does exactly whatever you think it does. To make sure it does, check the full paper and pretty pictures here.
In September/October I went on a month-long research visit to the Bergen Visualization group in Norway, which was great for several reasons:
I met soooo many cool new people as well as cool people I knew from conferences before. It’s really an excellent group in all ways possible.
Bergen itself is really heaven on earth. It has it all, mountains within walking distance, a harbor, waffles and lots of metal. Also, VCBM 2016!
A little more on these mountains…. I’m not much of a sporty person, but on my first weekend there, I was invited to hike up Ulriken (only the highest of the Seven Mountains they have, luckily):
Talk about life-changing experiences… Mind=blown by the view, experience and sheer exhaustion.
It’s quite addictive really. I hiked up there once more during my stay. I could definitely see that becoming sort of just a thing to do on the weekends while living there.
I presented at a medviz seminar, check the flyer here. Yes, there was a flyer with my face on it!
I got some great PhD advice and started collaborating on a paper together. I can really recommend a visit like this, if it is at all possible, to anyone doing a PhD.
2015 was definitely the year of collaborations. Good ones too (for me at least ^^)! I worked with people from Leiden, Magdeburg, Bergen and recently Koblenz, and they are all awesome and I hope to do more of that in 2016.
Then for the looking forward bit:
I hope to have more awesome collaborations in 2016.
I have approximately a million, ok four-ish, papers to wrap up and then…
I don’t want to alarm you or anything, but 2016 could be the year I get my PhD (correction sent in by cpbotha: get doctorified) . After which I’ll have to change the subtitle of this blog into something yet unknown. I’m not the world’s biggest fan of change, but let’s just say, ‘it is time’. I’ve been walking around at the TU Delft since 2005 (yes, really…), first as a bachelor student, then master, then PhD, and a decade is more than enough for me. I’m looking forward to starting something new somewhere else. Anywhere else 😉
Alright, I just spent my full two week holiday working interspersed with family visits, the first part of which is not really my style, but hey, desperate times, desperate measures. So I guess I’d better go finish 1 out of those million papers. Till next time!