SSH-WEB

[Jepzter]

Hello!

About
I really wanted to do something fun and that you could make use of, so I came up with the idea to connect to SSH remotes through the web and have all the devices connected to your account in a dashboard, so you can easily connect to them. I also found it useful to store data and access that for each devices that you have, just to know what you did last time or what went wrong.

I started this project the other week and I have start to make progress in the frontend and backend. The Frontend is build for simplicity and I hope you think so to, the backend is build as a multi-module project and the API is a standalone module, which means that the API will be exposed and free to use while finished in your projects, if you want.

Demo
The stability of this will be measured as soon as the SSH service is finished and only the MVP components of it. I will deploy the services online for you to try it, and all errors and warnings will be capture and sent to elasticsearch so I can see the most prio issues. The backend service for the website will also be monitored through elasticsearch and kibana.

Technologies used

• React
• Java
• Bootstrap
• Redux
• JAX-RS
• Netty
• Jdbi
• Liquibase (database version control)
• Elasticsearch
• Kibana

Technologies that might be implemented

• RxJava
• RxNetty

Whats to be done according to the MVP

• SSH Service (currently in build)
• Login system in frontend
• Basic settings page for all devices (very basic)
• See history of commands that was executed on the server (can be disabled by basic settings page)

Services (backend)
webapp-service
ssh-service
email-service
Proxy-service

Images
API:

Frontend:

 

[griimnak]

This is a good idea dude!
I think it'd be even better if you could incorporate scp aswell so you could manage files from devices saved

 

[Ecko]

You must be registered for see links

 

[Jepzter]

I knew about xterm, but want to build my own service that has everything ready to go after you've created an account.
 

This is a good idea dude!
I think it'd be even better if you could incorporate scp aswell so you could manage files from devices saved

Cool idea! That would probably be the next step to create a javaservice for that!

 

[Weasel]

How are you tackling the storage of connection data? Services like this include storage of very sensitive things.

 

[Jepzter]

How are you tackling the storage of connection data? Services like this include storage of very sensitive things.

No connection data will be stored other than ip and port not username or password for the device. The data of commands that has been executed can choose to be store or not be stored. If stored, all commands will be encrypted, probably AES. I think of a setting to store data up to X amount of days, 1 hour, 1 day, 7 days, 1 month, forever. However, this will include a way for the user to download a file (PDF, or a textfile will probably fit the best in this case), with the commands that has been executed and is after that removed from the database. Also, the IP and port will be encrypted as well. Yes this will cause alot of work for the backend service, but I think it will do for now with probably a maximum of 5 users

Another way to be tackle this would be you to connect to the service with the client secret, auth token and the target device UUID created, this will not do much with the storage of ip and ports, but the information is not public in the frontend and therefor never possible to fetch it from the API.

Do you have any better idea to tackle this?

---
Im currently working to get this up and running for open source with you being able to do pull requests.
Im pushing hard now to get the code readable, and is currently working on the SSH feature. Probably done with this in a couple of days.
 
Doing connections through SSH seems to be working as it should be. Though, the question is how the communication of this should go, I still think of using REST with sending commands. Why? well, we are do not really care if the connection is open all the time, the only thing we are concern about is receiving the stdout from the server, otherwise we could use some socket technology, but that does not seem neccessary in this.

What will be important for this microservice though is that it uses RxJava, so that we can do multiple commands at once for many different users. Otherwise I would probably have to scale this up a bit with setting up multiple services with loadbalancing as there can be delays of the stdout.
 
So, I've managed to setup so that its possible to run SSH commands through the Java service, and it looks like this.

You must be registered for see links

This run the command ifconfig to the server, and catches the response, now I have to setup an api endpoint to communicate with this service and then do the whole thing in RxJava, this will be a small task so it will probably go quick, not to much happening here. The question is if the main service should act like a client and route to this service or the frontend should communicate directly. I don't believe in running API request directly to the server so I will do a small service that will route traffic between the services. After this I will start doing the API Authentication using 0Auth2, and do the login system. I know this is not MVP to do the login system right now, but I feel like it should be done now anyway as it CAN actually be used, though without all the encryption.

On the frontend right now, not much is happening other than that you can choose a device and go into its "terminal" which is now just a regular styled bootstrap panel, so not much to see yet.
 
So I've actually done some updates on this, and thats mostly on the authentication part, because I though that it would come in handy as I said in previous post.

So, Im a fan of keeping stuff open, and therefor I've added so that you can integrate with the API using the endpoints, not only from the website but also from any curl request or anything other that can do a request. So to keep this "semi-secure" I've made so that yes, it is required for you to sign up on the website like any other user, but when you sign up you also get a client secret. With this client secret you would make a POST request to the API with your client secret. When you do that you receive the access-token and the time it will expire. Demo of this below,

You must be registered for see links

If something is wrong with the tokens/time,

Now with this, I could create a super-basic "getting started" login system which does not do much than log you in,

You must be registered for see links

If you are not logged in, you will be redirected to the login screen,

You must be registered for see links

The session storage will probably not contain these stuff, but for now it does, and I will change that later.
 
Now we are on the way really, now its possible to execute commands through the api and it looks like this, and responds like this. The URL should not be like this, in the end, hehe.

 
Well, I see a really big design flaw I did while designing this, what if you would like to use vim? You can't do that realtime with an api... I have to migrate this to an socket, I will probably be done with this in the weekend. Stupid me....
 
Sorry for inactivity. I currently thinking of doing a big refactoring do help scale the website and all other microservices around it. I will hopefully this weekend start implementing Apache Kafka, for all microservices to share specified topics around. Therefor everything can also easily be split up in small portion databases. What do you think of this?
 
Currently building the Kafka process. I actually writing a whole framework for building microservices easy, and I hope that this will help this forums development. While this is still in its early state I will describe how you will be able to use the Framework to work with kafka consumer and producers. I figured that setting Kafka up in a java service without Spring is not easy, and it's not that many tutorials about it, so this is the early state of the framework that now, only has support for handling Kafka.

The process is very easy, to throw events out into the Kafka stream you will use a stream publisher,

 EventPublisher.sendEvent("user-login-v1", user);

Notice that the user object in this case, is a serialized JSON Object, using Jackson. EventPublisher.sendEvent(topic, jsonObject) is a static method.

To make use of the events you've sent, you want something to listen to the event, in the same application, or another micro service, you declare them using the

@EventConsumer(topic = "user-login-v1")

annotation on the method.

Its very important that the EventHandler class extends the abstract class EventHandler, that will give you a handle method.

public class TestEvent extends EventHandler<User> {
   
    @EventConsumer(topic = "user-login-v1")
    public void handle(User eventData) {
        System.out.printf("Handled event for %s, with data %s", "user-login-v1", data.getTopic());
    }
}

The handle method will be the place where you call you application different services depending on what you will do with it.

All of this happens behind the scenes, so I really hope, when this framework is done you will have great use of this!

Any inputs about what you think of this @Holmes ?

 

[treebeard]

Keep up the good work!

You are one of a select few around here capable of turning out some quality work

 

[MayoMayn]

Damn, looks good!
Is this a private project, or are you gonna put it on git?

 

[Jepzter]

Damn, looks good!
Is this a private project, or are you gonna put it on git?

Thanks!
This is a private project that I have on bitbucket. But when it gets closer to the end, I will make it public, also this framework will have some of its functions public when its done, some are internal for my company. But for instance the Kafka package will be public.

The thing I have left now is doing the websocket instead of the API, though, I need the Framework to be part-done, because this is a project that really make good use of Kafka, as it is build upon microservices.
I hope you can find usage of Kafka as it is a very powerful ecosystem for all your application and services, even for the frontend.

Also, this is a video of the application posting events, and handling them. This also works if you from the docker container send out those events, it will handle them.

You must be registered for see links

I will also make sure that RxJava works better than it does now, doing async stuff should create so much mess in the code, I will make so that the Framework will handle the invokes by it self, onSubscribe, onNext and onError. When that's done I will do the elasticsearch part of the framework that will do logging in you application. This is the 3 core parts that will be realeased and will make a huge impact on THIS projects future and to be scalable.
 

Keep up the good work!

You are one of a select few around here capable of turning out some quality work

Thank you! Hope people find the end product to be very well developed.

 

[MayoMayn]

Just to mention, if you're going to do async stuff, you should do it inside componentDidMount and not componentWillMount as it gets called multiple times and the rendering is most definitely finished before the async stuff is.
Referring to your redux function that looks like a HTTP request inside your DeviceDashboard component.

E.g

class DeviceDashboard extends Component {

    componentDidMount = () => this.props.fetchDevices();

    // ...

    render = () => (
        <div className="dashboard">
            <div className="container">
                // check the loading property instead of the devices length
                {this.props.loading ? <h1>Loading...</h1> : this.createDeviceDashboard()}
            </div>
        </div>
    );

}

and then set the loading property as true in the initial state

You must be registered for see links

 

[Jepzter]

Just to mention, if you're going to do async stuff, you should do it inside componentDidMount and not componentWillMount as it gets called multiple times and the rendering is most definitely finished before the async stuff is.
Referring to your redux function that looks like a HTTP request inside your DeviceDashboard component.

E.g

class DeviceDashboard extends Component {

    componentDidMount = () => this.props.fetchDevices();

    // ...

    render = () => (
        <div className="dashboard">
            <div className="container">
                // check the loading property instead of the devices length
                {this.props.loading ? <h1>Loading...</h1> : this.createDeviceDashboard()}
            </div>
        </div>
    );

}

and then set the loading property as true in the initial state

You must be registered for see links

Thanks, I Will fix it!
I recently started working with react, redux and frontend overall! thanks for improvements

 

[MayoMayn]

Thanks, I Will fix it!
I recently started working with react, redux and frontend overall! thanks for improvements

You're welcome.
If you got any questions, don't hesitate to PM me or create a thread

 

[Jepzter]

So I guess it's not time to post an update really, but I will do it anyway even though its small. So the bug with eventlisteners freezing the application (single-thread applications sucks) is now fixed with RxJava. There is now an email module aswell, might not be so important maybe but its there and its super easy to use. A nice way to use the email function would be to listen on events like "user-registered-v1" and on that event send an email notification to the user with the text, "Welcome"!

The development I start with now is monitoring the application with elasticsearch, while I think of this I will probably make so that either it will log automagically on thrown exceptions or, I will create a special exception type for that to gather all information, and send it away. The coolest way would be the framework listening to all kinds of event, but im not sure thats v1. Some of you may wonder, why would we setup a elasticsearch server with either kibana or Grafana.
Well, firstly, you can monitor your application live, from any web browser, and see graphs of what ever you want to monitor on.
Secondly if you were to host, ugh lets just take a horrible example of a habbo retro, which produces alot of information in logs, you might miss stuff in the console, and having a hard time to find something that really diff from the normal behaviour, well now thats all handled and you see al strange things instantly, without being on the server.

The full RxJava part will come after this, why I want to wait with this is because Im not sure how I should do it yet, after that's finished I can keep on putting the last pieces of the MVP for SSH-WEB together, as it has all it needs to fully function.
 
No further updates now util the beginning of August, due to semester break from the real job and as well with this.
Have a great semester everyone!
 
Just started development again, still working on the framework and probably will for a week as I plan to use it in other projects as well. I let you know when the kafka is 100% done. When it's done I will also release parts of it here.
 
Kafka is already quite healty right now, it now takes in your POJO by extending the EventHandler. This all works with Jackson doing the magic in the background transforming the POJO to JSON and back. This also means that you just send the object in the eventPublisher. This is a GREAT way of building microservices and event driven designs.

EventConsumer

public class KafkaTest extends EventHandler<User> {
 
    @Override
    @EventConsumer(topic = "user-login-v1")
    public void handle(User event) {
        event.getUsername();
    }
}

EventPublisher

User user = new User("[email protected]", "TheOvster");
EventPublisher.sendEvent("user-login-v1", user);

You must be registered for see links

Actual kafka working.

the framework now comes with handling mail, you just add your email config to your YML.

The framework will also come with PDF generation, this is to make invoices, recipe and other stuff. Why you may wonder, well this framework is aimed for enterprise development, so I hope this come in handy. I update you all on this. I will make the project open source when the code has been refactored. We all know how the code looks when you try stuff the first times. Then the development of the main project will continue