Importing Large Number of Points from CSV with x,y,z coordinates to independent points

Hi everyone. I've seen a discussion about importing key-frames and some brilliant work-arounds people have implemented. Action Pro would give a method now to import CSV as key-frames. But not as individual and independent points.

I have some astronomical simulations and animations I do for education and science at an astronomy research institute. I have access to a database where I can grab an x,y,z coordinates (actually polar, and I have to convert).

Any idea if it's possible to use a CSV data file (let's just say 100 points) and import each as a point, wherein I can then parent a "star light" to each of them?

Honestly, if I could pull that off, that's something I would be willing to share. It exists in custom software here, and some other video software that cannot export those values into HitFilm (go figure).

That would be ingenious! I am using it as a background as opposed to fly-through. It would be freaking incredible to be able to pull in 1000+ objects as background stars (if anyone knows what GAIA is).

Otherwise, I create an artificial star system background either from a 2D image or particle simulator. I have a number of animations I'd love to do that for. If anyone has a way, I would post a sample created in HitFilm.



  • Triem23Triem23 Moderator
    edited December 2019

    @spydurhank just in case Filmer somewhere uses CSV, or if you have thoughts on converting CSV to nulls.

    @NormanPCN, @Stargazer54 @NxVisualStudio for your general knowledge and expertise. 

    @JoshDaviesCEO because this might inspire a change to Action Pro AND I think you'll be intrigued by this combination of science and Hitfilm. 

    @DannyDev cuz I think Josh might be intrigued by this and start talking to the Devs and Cedric and Ben have gotten all my tag love lately. Figured you're due. 

    @FilmSensei @GrayMotion @tddavis @JavertValbarr cuz I think you'll all be fascinated by this like I am. 

    @Strings123 out of my knowledge base, but I've tried to tag in those who can help. 

  • Let’s make this more exciting.

    For anyone who has followed Kepler’s survey for exoplanet detection, if I had star locations, I could effectively add exoplanets parented to the star, which I could, of course, then warp an image. 
    The people I work most closely with are galaxy astronomers. i might as well make it orbit  

    I cannot imagine importing 18 million mapped stars. Would t that be fun!

    Maybe that’s why some of our folks write their own software and scripts. 

  • Ooh, ideas!!! :) Thanks @Triem23 . :)

  • @Strings123 that's exactly why you science guys write custom code. General VFX software just isn't designed for that kind of hardcore simulation. VFX is about looking cool and plausible, not about actual accuracy. 

    Most space scenes are cheated like hell. Babylon 5 is the only show I can think of that set up a sector of space at 1:1 scale, from the ships to the station, planet, jumpgate and star... And when Foundation got released at the end of S3 and Netter Digital took over? Those guys messed all the scales up.

    Note a Hitfilm camera has a maximum draw distance of100,000 units, so you have an effective "hard limit" on your sim size. Let's say Sol=1 pixel. Ok so the max draw distance of the HF camera is a bit under an AU (and Earth is REALLY small at that scale). A truly accurate galaxy is probably beyond Hitfilm.

    I have a shot in-progress (on pause while I'm 5000 miles away from my Hitfilm computer) where I am using a scale of 1px=3 miles, which is just enough to let me put the camera between a 1:1 scale Earth/Moon system and see them both when the camera rotates... It's also enough scale for my Death Star (but my Death Star is a bit larger than "canon" Death Star II size at an even 1000km diameter - - yes, I'm mixing scales. I'm American, I THINK in Imperial, but will do the conversion, as needed, for metric).

    It's going to tricky when the debris of the moon goes streaking past the (rotating) camera to slam into Earth... I might have to cheat things since my "Dino killer" debris is only about 15-16 px wide...

    1:1 scale in VFX is a bitch. There's a reason it's usually faked. I mean Earth's atmosphere at this scale is about 34 px. Not much room or time to do flame trails before impact.... 

  • Stargazer54Stargazer54 Moderator
    edited December 2019

    @Strings123 ; Wow.  Totally awesome project!   But I have to say I don't think there is a way to directly import a CSV file into HF.  @Triem23 might be on the right track with a potential input through Action Pro, but that would be something for the development team.

    However, you can bring a CSV into a 3D package and use the data points as motion data.  I've done this with Lightwave on a previous project.  @spydurhank might know a way to do that in Blender.   Once imported, he might be able to get the data into HF.   

    But it sounds like the project you are describing is bringing in "position" data (x,y,z) for individual points.  That is something entirely different than motion data to move a single object across the screen.  In HF each point will be its own layer so it could get very cumbersome for 100 points.  

    Now that I think about it, you could build a 3D .obj file with your CSV data.  Essentially placing a "v" in front of each data row (for x, y, z) would make each row a point in 3D space.  An obj file is an ASCII file that is readable into HF as a 3D object.  That would give you the 3D point cloud you might be looking for.  It would all be one object layer, too.  To put a flare (or light) on each point might be difficult.  But if you wrote the obj file such that each data point made up a polygon then you might be able to have that appear bright enough to attach the flare to.

    I think it might be easier to wash the obj through a 3D package that could instance a polygon on each point location and then bring that into HF.  Again, I think the key is starting with a readable obj file made from your data.

    Below is an example of a simplified obj file with just 3D points.  However this doesn't read into HF because HF wants the "f" parameter that defines faces derived from the points.   So you would have to wash it through a 3D program to instance polys onto the points.

    #    OBJ File Generated by LightWave3D
    #    LightWave3D OBJ Export v2.3
    o cube-test-points.obj
    mtllib cube-test-points.mtl
    g default
    v -0.608096 -0.414341 1.23469
    v -0.188258 0.416453 0.486075
    v 0.422512 0.2121 0.554843
    v 0.583778 -0.460089 0.306924
    v -0.043027 -0.749391 -1.09732
    v -0.822238 0.209423 -1.06993
    v 0.1554 0.328275 -0.558
    v 0.389216 -0.378874 -0.418941

  • edited December 2019

    Agreed - absolutely. And that's also for more scientific use. @Triem23 thanks a ton for the feedback and discussion. It is quite impossible to do a galaxy in HitFilm. But it's fun to dream! (Side note, in addition to custom software, they are now committed to Houdini because it also has massive import/export capabilities, allowing them to write code where they need to.)

    My animations are much lower in complexity. I'm not simulating. I help scientists convey conceptual aspects to public or non-astronomer community. So my complexity is much lower and not research. Is slim and quicker. By the way, I don't work in the science group, I work in our IT division and do this "on the side" for some scientific friends here. I have a lot of public contact and just love doing this stuff. 

    I'm not trying to go too far with it - but my mind wanders just like other creative people in this awesome forum.

    Right now I have a planetary formation region with accretion disk (dust collecting) around a newly formed star. I run the camera through and look back and then show some polarized light animation to show how its polarized and how it's used.

    I have it completed and did it all in 5 hours. Now, I have to add some labels and a star background. (Although the Clipping Mask messed with me, until I just got rid of the element that threw it at me).

    So - thanks for 100,000 units info. If I imported stars, it would be limited to a hundred or so, maybe a couple hundred - enough to create a background, and they would be kept very small. My present composite scene is 3,000 radius - enough for what I need. You have to cheat. So, for the sake of scale, 3000 pixels = ~ 100AU = 100 x 93m miles =  9.3B miles.  (Pretty off the cuff estimate)  The star I have is WAY out of scale - it's for illustration purposes. And, of course, the fly-through speed is on the order of 12B miles in 10 seconds. Not a bad speed. (930m miles per second, or roughly 3.5 trillion miles per hour. Speed of light = 671,000,000 miles per hour).    100,000 units represents a pretty good distance. Like what, 307 billion miles?

    MARTY.......We've passed the speed of light! We're gonna need a few more gigawatts!

    I also would probably limit the quantity of stars to the visible regions.

    In the field images warped inside the surface of the solar system snow globe.

  • edited December 2019

    @Triem23 ; Thanks for the tag in.   While it is way far beyond my knowledgebase for writing script and such I do find it extremely interesting (much like that Reallusion program I posted about today and I looked at Ebsynth a few years ago, but it appears to have made some significant jumps...and it's free!!!) and will follow progress happily. 

  • Stargazer54 

    Thanks a lot for the input. If you wash it through a 3D editor, and it provides the face data, then it's all a single object in HF? Although the number of points would be the same. HF would interpret the collection as a snigle object made of a lot of points. So if I literally could take the nearest 1000 stars, create an object form that data, then it's a matter of displaying those points.

    My next personal development steps is to step into the world of 3D object creation using imported data. I know little about those programs. I've only done a ton of 2D CAD during my design years (and a LOT of software in my life, but not scientific software). 

    I don't have a budget. I'm wonder if there's a 3D tool that could import CSV based on a defined import (kind of like Excel) that I could create objects. IN FACT - Even if it did have face data, it might be possible to keep that object with hidden faces, but the points being visible might not be supported? Maybe you'd have to do each one individually. Also - note that stars are different brightness and colors - that's going WAY FAR. Good lord, I could get lost in this.

    After this project gets published, I'll put it up somewhere. It's not nearly as fancy as it seems. If it's taken me 5 hours to get this far and learn, it's probably a 1-hour project for you guys. It's a learning project for me.

  • Triem23Triem23 Moderator
    edited December 2019

    As a side note, this short video shows some interesting things about 1:1 scale vs faking it. The upper left version is fake scale where I just had the "looks good" Earth-Moon fake recede into space. Then I saw video from the HELIOS probe shooting from that narrow window of distance where the Moon can eclipse Earth. 

    So I rebuilt the shot to accurate scale and distance, and was really surprised at what happened. It doesn't "look right" because it doesn't do what we see in movies, but it IS right because it's all at accurate scale and distance!

    Second side note. This is also when I figured out Parallax and Caustics render lights independent of 3D layers/models. The up left model is "double lit" as a model and its parallax effect. Lower right turns the model "illumination" off, but the Parallax effect has illumination on... 

  • Stargazer54Stargazer54 Moderator
    edited December 2019

     @Strings123 ; You can't really make faces (polys) out of random points because you would wind up with some sort of crazy shaped polys.  You need to keep the data points separated and use them as reference to place a circle a sphere on.  Here is what I came up with after instancing spheres on the vertex data in the previous example:

    Also, here is a good site explaining the obj format: Click on the link at the top of the page for an example of how to connect vertex data to faces to make a box 1 meter square.

    But I think what you want is a point cloud that can be used to instance objects onto.

  • Stargazer54Stargazer54 Moderator
    edited December 2019

     Here is what the obj file looks like read in to HF and applying flares:

    @Strings123 I would be happy to gin up an obj file for you based a sample CSV file if you like.

  • @Stargazer54 if we're talking thousands of small models, could one get away with tetrahedrons, or are you using icosagons as "Low res spheres?" 

  • I can do that in Filmer with a little scripting, no problem at all. I'd just need to be able to import the csv file into Filmer for conversion to either and or both 3D points/empties and simple object primitives. 

    Scale doesn't matter as we'd have to do things within the confines of Hitfilm's 3D space and the already mentioned  camera clipping all of my Filmer exporters already take care of scale so it'd be pretty easy. Hitfilm can only import 50 Empties/tracking markers at a time so we'd also be restricted by having to do multiple exports but that won't be a problem. I can already export entire 3D environments and assets to Hitfilm. :)

  • @Triem23 ; Absolutely.  Anything low-res 3D will do.  I was thinking "spheres" so that it would be visible from all angles.

  • Filmer = looks like I've go to look into that. (What's the full name of that?)

  • @Stargazer54 ; @Triem23 @spydurhank

    The stars would be so far away (forget scale for now - let's just assume "out there"), they would essentially be points, but I guess you have no choice but assign a radius.

    A single visible point from say, 20,000 to 100,000 units away.

    Imagine looking out form our sun and seeing background stars. That literally is it. They are "icing on the cake" for relative to the core content.

    But now that I'm dreaming.....forget it. Low res would do.  If I were to fly in to anyone of them, I'd model it specifically.

    You guys are awesome for chiming in.

  • edited December 2019

    @Strings123 ; Filmer is a souped up version of Blender that is @spydurhank 's project.  He has done a lot of coding to make possible to transfee files or scenes between Blender and Hitfilm and, I think he is branching that into other programs too.  He posts WIP photos of his model work in his User gallery post:

  • edited December 2019

    Hey @Strings123 ,

    Just Filmer, :)

    It is a custom build of Blender 2.80 specifically to export to Hitfilm Pro but my export concepts actually work in any 3D application and render engine.

    My exporters take care of scale so that part doesn't matter because they scale Filmer 3D units to Hitfilm 3D units. If you guys saw my first live chat with @FilmSensei and @GrayMotion a few months back, I basically do what this thread suggests except I was exporting an animated Particle System. :)


    Whoopa… thanks @tddavis . :)

This discussion has been closed.