Clickhouse jdbc schema [Airbyte integration] - java

I'm really looking for and advice on how to set up data streaming between Oracle db (just for example) and Clickhouse database using Airbyte ETL tool.
I've created the source and destination connectors and successfuly tested the connections but now I'm facing the issue in syncing the data. While the source is sending the data correctly, the destination (Clickhouse) JDBC driver requires the schema to accept those tables.
So, I don't know how to retrieve this schema and where it should be stored. Maybe there're some means to create this schema, because when I inspect the log it displays such thing:
So, basically, the error is connected with Exception in thread "main" java.lang.IllegalStateException: jdbc destinations must specify a schema.
Please I will be really grateful for providing any means for resolving that issue!

I think use should try version 0.1.1 of ClickHouse destination. It fixes your issue https://github.com/airbytehq/airbyte/pull/8982

Related

Connecting to data source without specifying database name

I'm using spring boot and database mongo. Our database is placed on AWS cluster. Inside data source we have couples databases(AccountsDB, SubscriptionsDB etc). I did realization, when in Spring boot I connected to 2 databases using disable autoconfiguration in application.properties file and connect to mongoDB twice: ones, to database AccountsDB(
mongodb://${MONGO_DB_USERNAME}:${MONGO_DB_PASSWORD}#${MONGODB_HOST}:${MONGODB_PORT}/{AccountsDB}?replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false
), and second connection to SubscriptionDB(
mongodb://${MONGO_DB_USERNAME}:${MONGO_DB_PASSWORD}#${MONGODB_HOST}:${MONGODB_PORT}/{SubscriptionDB}?replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false
).
But my manager unhappy with this realization... There is any way to connect to data source without clarification of data base name? I couldn't find any solution for this, if we are not clarify its not showing any data. Thank you for any replying
I tried to connect without clarification, and it doesn't display any data
mongodb://${MONGO_DB_USERNAME}:${MONGO_DB_PASSWORD}#${MONGODB_HOST}:${MONGODB_PORT}/?replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false
Might be any way to connect to datasource and after using SpringJPA make connection?

How to create a database if it does not exist with springboot/JPA and SQL Server?

I am surprised I haven't found an SO question that answers this. I am trying to connect a springboot/JPA application to an SQL Server on my local machine. I have the application setup so that it can connect to a database if it it exists, but if I change the JDBC URL to create the database if it doesn't exist then it fails. Here is what the properties look like when it fails.
spring.jpa.hibernate.ddl-auto=update
spring.datasource.url=jdbc:sqlserver://localhost:1433;databaseName=testing;createDatabaseIfNotExist=true;
spring.datasource.username=hello
spring.datasource.password=Hello1234
spring.datasource.driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver
spring.jpa.database-platform=org.hibernate.dialect.SQLServer2008Dialect
spring.jpa.show-sql=true
Here is a snippet of the error I receive when starting the app:
com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user
'hello'. ClientConnectionId:971a3369-258b-4713-bddc-cda559b9fe94 at
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:262)
~[mssql-jdbc-8.4.0.jre11.jar:na] at
com.microsoft.sqlserver.jdbc.TDSTokenHandle
If anybody has any thoughts as to how I can change this so the database is created if it does not exist I would be very thankful. Thanks in advnace.
I don't think a database can be created using JPA.
It has to be created manually or in some other ways, but JPA won't do that for you.
And it would be a bad practice as well to create the database using the application itself, and the use of same credentials.
Yes, definitely you can auto-create a database with JPA for that
spring.datasource.url=jdbc:sqlserver://localhost:1433;databaseName=testing;
createDatabaseIfNotExist=true;
line should be changed to:
spring.datasource.url=jdbc:sqlserver://localhost:1433
/testing?createDatabaseIfNotExist=true
In practice your application should never create your database so its not really a problem most of the time(Outside small databases like sqlite3). Different databases would handle this situation differently as well.
In your case I do not see this as a valid jdbc parameter in the documentation.
I would recommend creating the database in advance with a privileged user separate from your application user.

Get JDBC connection working in play framework (java)

Was wondering, what steps am I missing to get a jdbc embeded h2 database working in my play application? Following these docs.
So far editted Application.conf file to contain this:
db.default.driver=org.h2.Driver
db.default.url="jdbc:h2:databases/test"
db.default.user=test
db.default.password="testtest"
Next I created a libs directory and added the jar file
h2-1.3.174.jar
Is this necessary or does the provided driver handle all types of h2 databases (embeded and server - I know it handles in memory)?
Now in the controler how can I access the database? Do I have to start/shutdown the database?
I know I can get connections from the getConnection() method in play.db. But everytime I execute a statement through this connection I get an exception saying no data is available. If I then check - looks like directory
databases/test
was not created so no database files exist.
What am I missing?
H2 works out of the box. Just create a new project in the terminal.
Otherwise, to your listing:
I think you should change db.default.url="jdbc:h2:databases/test" to db.default.url="jdbc:h2:mem:play"
don't need to create lib directories. It's all handeled by the build in dependency mgmt sbt
Just use the model objects and call save/update. No need to call start/shutdown
you are in a framework, it's all there ready for you...
I think you should start reading the documentation from the beginning to the end and examine the example applications. It's all there what you are looking for.
In addition to myborobudur's answer I'll only mention, that you don't need to use memory database, as you can for an instance use file storage (Embedded) or even run H2 as a server and then connect to it with TCP in Server Mode... Everything is clearly described in H2 documentation.

Issue when deploying application to GlassFish Server - mapping issue?

I'm trying to deploy an application to my GlassFish Server environment. I've set it up so that GlassFish creates a connection pool to a postgreSQL database on another server (not localhost) where the database is located. I test the connection and then try to deploy the application. It fails with a java.lang.RuntimeException: EJB Container initialization error, and my error log contains the following: http://ideone.com/UlZXut (put it here due to its size). There were other warnings above these, but they only referred to tables already existing.
As according to this, I thought that the required sun-cmp-mappings.xml file (the one I assume would be necessary for this correct mapping) would be automatically generated upon deployment, but it seems I was wrong. Could anyone shed some light on this situation?
My apologies if this is not the absolute best part of SE to post this, but it is related to development tools and I did see a number of related posts.
Your error log indicates that you are trying to create table(s) with DOUBLE as a datatype. In Postgresql, that datatype is actually called "double precision". What happens if you revise the table definition to use "double precision" instead?
on startup Glassfish tries to create the DB tables for your java code. It fails to do that and it fails to startup.
Check the configuration of your ORM mapper.

Recovering HyperSQL Database

We are using JBoss 4.0.3 SP1 using Hypersonic as its internal storage engine (timers, queues, etc.)
The database is no longer accessible (most likely data corruption) giving error; Caused by: org.hsqldb.HsqlException: old version database must be shutdown.
Is there a way or command to shutdown the database to recover the data (~150MB) stuck in the flat file?
Edit
Resolved the connection problem by downgrading to Hypersonic 1.8. How ever the database itself is corrupt. You have rows with duplicate keys, and lots of null fields. Having read this article and this one, i'm in the process of changing the default JBossMQ database to MS SQL
According to the hsql documentation the command is SHUTDOWN. As far as I remember, hsql saves the data as a set of sql queries, so you need to have a look on the file.
Notice that according to JBoss,
Hypersonic(HSQLDB) SHOULD NOT be used
in production at all. It is not
suitable for production use.

Categories

Resources