Im tryinig to configure the BigQuery adapter for Oracle golden gate (12c). But unfortunately Im not able to understand this gg.classpath. In the doc, they mentioned to download client Lib from the GitHub
But it contains so many folders, I don't know what folder I can download and where to copy them.
Also we have download some JAR files, and put it a directory.
Next, download the following JARs from Maven Central, and then include them in the classpath for the BigQuery Handler:
Where I can put these JAR files?
Can someone give me an example to configure the gg.classpath
Oracle doc link: https://docs.oracle.com/en/middleware/goldengate/big-data/12.3.2.1/gadbd/using-bigquery-handler.html#GUID-A8426075-81E8-47AB-BA15-3C564751A490
example gg.classpath: gg.classpath= /path_to_repository/bigquery/libs/*:/path_to_repository/bigquery/*
You can download the jars file from maven as mentioned in the documentation. Once you have download the jar files you can place them in any directory of the server/machine where you have installed Oracle GoldenGate for BigData. Lets say for example /u01/OGG. After that you can include that directory in the classpath as below.
Example:
gg.classpath=dirprm:/u01/OGG/jdk1.8.0_221/:/u01/OGG/oci-java-sdk/lib/:/u01/OGG/oci-java-sdk/third-party/lib/:/u01/OGG/hdfs_jar/:/u01/OGG/parquet_jar/*
In the above example i have downloaded OCI Handler, Parquet Handler and their dependency and placed them in path /u01/OGG/. Here handler_name is oci-java-sdk and parquet_jar.
Just leaving the answer here, so others can benefit from this.
-- Steps for manually build the jar files
-- Or download all the jar from the above link and skip all these steps.
apt install maven
git clone https://github.com/googleapis/google-api-java-client-services.git
cd google-api-java-client-services/clients/google-api-services-bigquery/v2/1.29.2/
mvn clean install
-- Copy all the jar files to /opt/ogg/jar
mkdir -p /opt/ogg/jar
cp target/*.jar /opt/ogg/jar
-- We also need to download the following JARs from Maven Central
-- download path /opt/ogg/jar
api-common-1.6.0.jar
gax-1.28.0.jar
gax-httpjson-0.45.0.jar
google-auth-library-credentials-0.9.1.jar
google-auth-library-oauth2-http-0.9.1.jar
google-cloud-bigquery-1.31.0.jar
google-cloud-core-1.35.0.jar
google-cloud-core-http-1.35.0.jar
google-http-client-jackson-1.23.0.jar
guava-25.1-jre.jar
threetenbp-1.3.6.jar
Or I have a zip file that contains all the necessary files. You can get it from the below link.
https://github.com/BhuviTheDataGuy/medium-blog-files/raw/master/golden-gate-bigquery/jar-files.zip
For more detailed Step by step Setup for Oracle to BigQuery, refer the below link.
https://medium.com/searce/sync-oracle-to-bigquery-with-golden-gate-bigquery-adapter-59991bbdb5e3
Related
I am trying to create a launcher application in Java that can invoke executable .jar files. An easy way for this is to host the executable .jar files in a cloud instance and the launcher application would just download the .jar files from there into the user's local folder. The dilemma is the absence of the cloud service or a server that can host the jar files.
As a workaround, I thought I can just host the jar files in BitBucket and that is where the launcher application will download the executable files from. Though, I think this is a little unconventional since it is mostly used to version source codes (please correct me if I am wrong). Also, I am not sure if it would be possible to invoke FTP or download files from there (perhaps through BitBucket API, maybe?).
Another option is that it could be possible to make the launcher app to create executable jar files from the git repo and download the package executable jar files into the user's folder and invoke from there. If this is possible, I would appreciate any leads towards this option.
Thanks!
yes you can push your binaries to github (getting raw content is difficult from bitbuket your request need to have authentication and stuff).
here is how to for that
copying here as well in case get's removed.
How to create a maven repository for your github project step by step
Clone your project in a separate folder
(note: replace ORGANIZATION and PROJECT)
git clone git clone git#github.com:ORGANIZATION/PROJECT.git my-repository
cd into it
cd my-repository
Create a new branch (here named repository)
git branch repository
Switch to that branch
git checkout repository
Remove all files
rm -rf file1 file2 file3 .. etc
Install your jar in that directory
(note: replace YOUR_GROUP, YOUR_ARTIFACT, YOUR_VERSION and YOUR_JAR_FILE)
mvn install:install-file -DgroupId=YOUR_GROUP -DartifactId=YOUR_ARTIFACT -Dversion=YOUR_VERSION -Dfile=YOUR_JAR_FILE -Dpackaging=jar -DgeneratePom=true -DlocalRepositoryPath=. -DcreateChecksum=true
YOUR_JAR_FILE should point to an existent jar file, this is why it's best to create your repository branch in a different folder, so you can reference the existing jar in /your/project/path/target/artifact-x.y.z.jar
Add all generated files, commit and push
git add -A . && git commit -m "released version X.Y.Z"
git push origin repository
Reference your jar from a different project
The repository url you just created is > https://raw.github.com/YOUR_ORGANIZATION/YOUR_ARTIFACT/repository/
2nd option is also a good approach as you user will get always the updated content but that will add building time into it, depends on your use case.
In general, it's not a good idea to store build products in a Git repository. Generally build products are large, tend to compress poorly (especially JAR files, which are already compressed), and don't need to be versioned. This is exactly the behavior you don't want from your Git repository.
While you can indeed do this, you're likely going to find that Bitbucket is not going to want to host your Git repository for this purpose, and then you'll have to find an alternative solution. You'd be better off hosting these assets in a cloud storage bucket or even on a static server and downloading using HTTPS. If this isn't achievable, you need to consider why that is, and look into solving it.
If these JAR files are build products for a release of software you're already hosting on Bitbucket, then it's probably fine to use the standard release asset functionality of Bitbucket for that purpose. You just don't want to use a Git repository, which is significantly more expensive to serve than static assets.
You also don't want to use FTP because (a) it's slower than HTTPS and (b) due to people not complying with the spec, it's impossible to use TLS with it consistently. Your software will have an exploitable security bug if you download data over an unencrypted connection without integrity checking and then execute that data.
I have a java project loaded into IntelliJ IDEA. The code requires JDBC and so I downloaded and unzipped the sqljdbc_6.4.0.0_enu file into the recommended directory, "C:\Program Files\Microsoft JDBC Driver 6.4 for SQL Server". This creates the jar files:
mssql-jdbc-6.4.0.jre7.jar
mssql-jdbc-6.4.0.jre8.jar
mssql-jdbc-6.4.0.jre9.jar
in the directory, "C:\Program Files\Microsoft JDBC Driver 6.4 for SQL Server\sqljdbc_6.4\enu". And so, now, how do I make use of all this in my project in IntelliJ IDEA? And if I later put all my java code into Git Hub, how can I ensure that another user will have code that will compile and link? Will I have to include the .jar file in Git Hub somehow?
you should not do this. For your case, you can choose one of them:
create lib foder into project folder - put .jar files in the folder- setup module with library point to the folder - commit metadata folder of IDEA such as .idea, {projectName}.iml (not recommend)
The best solution is using maven or Gradle, you are able to put the dependencies which are neccessary into pom.xml file. (Recommend)
You can refer this for adding external jar files in IntelliJ IDEA.
As for GitHub, you can put your jar files there by removing *.jar from .gitignore configuration file. However this is not a recommended practice to have JAR files in repository as they've considerable size. You can consider using some build tool(maven or gradle) to add external dependencies.
I have a project where I want to add an external JAR file. The desired external JAR file has a nifty Github page with source, but no pre-compiled JAR file.
These are the steps I've completed so far:
1. I have downloaded the source in a zip. (its Twinkle from SwingFx.ch in case you're interested)
2. I have extracted the zip file to my workspace.
3. I have created a new project with the same name as the extracted folder from the zip file. (project loads the source successfully)
4. I select the export option from the File menu and selected the 'JAR file' option and clicked next.
Note: I had to add an external library to the above Twinkle project for it to build successfully (in case that makes a difference to the settings).
On the JAR File Specification page there are multiple check-box options available(see below):
Export generated class file and resources
Export all output folder for checked projects
Export Java source files and resources
Export refactorings for checked projects
Compress the contents of the JAR file
Add directory entries
I am not sure which are supposed to be selected and if it makes a difference in the behaviour of the project I will add the (soon-to-be) exported JAR file to. I tested it by exporting with the default settings. This worked ok.. However, I now do not know if I should have chosen different settings in case of any reasons I am not aware of. I am not sure if there are specific settings I should choose when I intend for the JAR file to, specifically, be added as an external JAR file to another project.
Please enlighten me!
This is a traditional Java library that uses Maven. It should be fairly easy to build using Maven, which should be better and quicker to build this, if you already have Maven and git installed.
Let's consider that you did not download the source file as a zip, but take the github approach, where you'd use git to download the source code.
If you don't have git, download its latest version and install it.
If you don't have Maven, download its latest version and install it.
Once Maven and git are installed, make sure the Maven and git binaries are configured in your environment PATH variable. If not set, you would, on the Windows platform and for Maven binaries, set it this way (using the default installation path):
set PATH=%PATH%;C:\Program Files (x86)\Apache\maven-3.1.1\bin
Create and change directory in a work directory of your choice, that we'll refer to %work_directory% from now on.
Run the following:
cd %work_directory%
git clone https://github.com/spreiter301/Core.git
git clone https://github.com/spreiter301/Twinkle.git
cd Core
mvn clean install
cd ../Twinkle
mvn package
6. Retrieve the twinkle-1.0.0.jar file in the newly created '%work_directory%/Twinkle/target' folder.
In this case, it was necessary to retrieve the Core library because it is a dependency of the Twinkle project. Normally, this is not necessary because dependencies are automatically retrieved from a maven repository. But in that case, that dependency is not available on any Maven repository. Hence we manually retrieved the dependency from github, compiled it and installed it in your local cached repository. Then we could package the Twinkle project into the JAR file.
This should do it. If you want a 5 minutes tutorial on Maven, there is a tutorial for this here. I highly recommend it, you will encounter this often in the Java world. Maven is the standard build tool for Java, just like 'make' is for C, 'rake' for Ruby, 'sbt' for Scala, etc..! Good luck with the rest.
I have created a java project using eclipse and in that project I have included a few jar files such as sqljdbc4.jar, jtds-1.2.4.jar, and log4j-1.2.17.jar. I have also included a dll for windows authentication in this project. I'm attempting to export this project as either a .zip .tar etc so the project can later be imported by someone else and work with the project.
I want to export the file in a way that the other developer can import the project and instantly begin working without having to download the jar files that the project is dependent on. Is this possible at all
I have already tried what i have thought would have worked going to export -> Archive file -> save in .zip format. I was unsuccessful with this I also figured going to properties -> Order and Export tab and selecting the needed jars would work and then exporting still unsuccessful.
Once the other developer imports the project the jars are not there for him...
I would use Ant. It is integrated right into eclipse and takes an .xml file called build.xml. It works by specifying "targets" in the xml file that each perform some action such as "build" (i.e. run javac), "deploy" (i.e. run the jar command), etc. There is a command that can be called from within the jar target called zipgroupfileset which you can use to package all of the jars within your main jar file.
More info about ant can be found here
There are many variations on the following but the basic idea is to
use a code library and share the project. The repository
can be local to you or even better put it on an http server
( ala VisualSVN or google or the like ).
1st (local) solution:
Install SVN ( http://subversion.apache.org/ )
Install SubVersion
Create a repo on your local file system
Share the project
Dump the repo
Send the repo
Load the repo (Other developer)
2nd (network) solution (using google code as your library)
Install SVN ( http://subversion.apache.org/ )
Install SubVersion
Put your project on GoogleCode (http://code.google.com/)
Other developer gets project
I started using Apache builder for a project and I am surprised how little code I need to build my project. But after the first successes, a problem arose:
I have some required text files stored in a jar file which can be downloaded from a repository. I have included it in my buildfile as an artifact and can download an compile with it.
Now I want to extract the contents of the jar and place it in the root folder of my project. Is there a way that Apache Buildr can do this for me? Just adding it to the classpath is not sufficient.
I believe this is the example you are looking for, from http://buildr.apache.org/artifacts.html
bean_jar = file('target/app/bean.jar'=>unzip('target/app'=>app_zip))
You can also directly call Buildr's Unzip, just don't forget the extract at the end.
Unzip.new( 'path/to/extra/to' => 'file/to/unzip').extract
A jar file is a zip file. You can use buildr's built-in unzip helper to unzip it wherever you'd like. The unzip helper is documented in the artifacts section of the buildr docs.
If you need to get the filename of an artifact that buildr has downloaded, you can call #to_s on the artifact:
artifact('net.example:text-files:jar:1.0.0').to_s