Yesterday I had to export all AEM tags into Excel file. While surfing for the best solution to do this, I've found out that almost everyone advices writing custom code that takes all of tags and enters it into Excel file.
I consider that solution good, but since there are a lot of people that do things like this for the first time and it will probably take some time for them to figure out how to do this.
For them, let's share some workarounds for this problem.
To get a comma separated list of tags I would recommend using the command line, the build-in AEM query builder, curl and jq (https://stedolan.github.io/jq/).
The general approach:
Use query builder to build a JSON representation of /etc/tags
Use curl to "download" the JSON
Use jq to "parse" the JSON and create a CSV
Example:
Exporting all tags below /etc/tags with their path, title and description would look like this:
curl \
--user admin:admin \
--silent \
"http://localhost:4502/bin/querybuilder.json?p.hits=selective&p.limit=-1&p.properties=jcr%3atitle%20jcr%3apath%20jcr%3adescription&path=%2fetc%2ftags&type=cq%3aTag" \
| jq --raw-output '.hits[] | [."jcr:path", ."jcr:title", ."jcr:description"] | #csv' \
> tags.csv
This would send a GET request to your local AEM instance (http://localhost:4502), authenticate as user admin with password admin(default settings for AEM), use the query builder API (/bin/querybuilder.json) to get all resources with type cq:Tag below /etc/tags and of those resources it would "select" the properties jcr:path, jcr:title and jcr:description.
The resulting JSON looks like this:
{
"success": true,
"results": 2,
"total": 2,
"more": false,
"offset": 0,
"hits": [
{
"jcr:path": "/etc/tags/experience-fragments",
"jcr:description": "Tag structured used by the Experience Fragments feature",
"jcr:title": "Experience Fragments"
},
{
"jcr:path": "/etc/tags/experience-fragments/variation",
"jcr:description": "A tag used by the experience fragments variations",
"jcr:title": "Variation"
},
]
}
Next, the command above will pipe the resulting JSON from the query builder to jq, which will use the "query" .hits[] | [."jcr:path", ."jcr:title", ."jcr:description"] to read only the hits array and of every item in that array only jcr:path, jcr:title and jcr:description. The resulting arrays are then used as input for #csv "string formatter" of jq, which will create proper comma-separated output.
The JSON from above would be formatted to:
"/etc/tags/experience-fragments","Experience Fragments","Tag structured used by the Experience Fragments feature"
"/etc/tags/experience-fragments/variation","Variation","A tag used by the experience fragments variations"
The last part of the command > tags.csv will just redirect the output to a file called tags.csv instead of the command line.
AEM has a query builder debugger which you can use to create queries that you then can use in your command line command:
http://localhost:4502/libs/cq/search/content/querydebug.html
The query parameters I used above would look like this in the tool:
path=/etc/tags
type=cq:Tag
p.hits=selective
p.limit=-1
p.properties=jcr:title jcr:path jcr:description
You can add properties as you like, but for them to show up in the CSV you also have to update the query used by jq.
If you add translations to your tags they will be stored in a property called jcr:title.<language-code>. If you translate your tags to German for example, you would have two properties: jcr:title and jcr:title.de. If you want the translation you have to extend the p.properties and add jcr:title.de etc.
If you need to export tags, here is simple solution on how to easy export them and then there are several ways how to import it in the Excel, without need to write custom code for this.
To export AEM tags do the following 5 steps:
Open package manager
Create package (give it some meaningful name)
Edit created package
Select "filters" tab
Enter path to tags you wish to export (for example: http://localhost:4502/libs/cq/tagging/gui/content/tags.html/etc/tags/geometrixx-outdoors)
you can find it under Adobe Experience Manager -> Tools -> General -> Tags
Done, Save
Build package
Download Package
Then you have all tags inside {download_package_name}/jcr_root/etc/tags.
Now there are several ways of getting downloaded tags to Excel file. This is how to do this on Windows -
Source: Is there a way to export a folder structure into excel?
Find the folder in Windows Explorer, then Shift Right click on that
folder and select "Open Command Window Here". Type the following
prompt:
dir /a /s /b > filelist.txt
That gives you a text file saved in your top folder, which you can
open in notebook, then copy and paste into an Excel document.
There's a GUI in AEM to export content as excel [well almost]. To get a tsv file of tags in the system AEM's bulk editor can be used. Navigate to '/etc/importers/bulkeditor.html' on the server. Set path as '/etc/tags' or a subtree. In query field type 'type:Tag'. Select the properties that need to be exported and hit search. The results can then be exported to a tsv file via the export button.
References
https://helpx.adobe.com/in/experience-manager/6-4/sites/administering/using/bulk-editor.html
https://helpx.adobe.com/experience-manager/6-4/sites/developing/using/reference-materials/javadoc/com/day/cq/wcm/commons/search/GQL.html
I don't have the privileges to comment on the accepted answer but thought it would be useful to add to the excellent accepted answer: If you're on >= 6.4 then you may need to change the path in your query to /content/cq:tags (1) or query both paths if your install has been upgraded over time and you have long lived tags. I was getting 0 hits even though I could see several hundred when viewing http://localhost:4502/tagging
(1) https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager/tagging-issue-in-aem-6-4/qaq-p/320994
Hopefully, this non-answer isn't overly frowned upon.
I am working with ESAPI to try and validate windows directory paths. For some reason, the part of my directory path named \14\ is getting converted into a CRLF. The error I am receiving is below, what am I not understanding correctly? I feel like my regex should be working.
WARN IntrusionDetector [SECURITY FAILURE Anonymous:null#unknown -> /project-test/IntrusionDetector] Invalid input: context=directoryPath, type(DirectoryName)=^[a-zA-Z0-9:/\\!##$%^&{}\[\]()_+\-=,.~'` ]{1,255}$, input=C:\UsersTESTUS~1AppDataLocalTempTestCase8002TempWorkSpace, orig=C:\Users\TESTUS~1\AppData\Local\Temp\14\TestCase8002TempWorkSpace
As you can see, I am using the regex:
^[a-zA-Z0-9:/\!##$%^&{}[]()_+-=,.~'` ]$
My input is:
C:\Users\TESTUS~1\AppData\Local\Temp\14\TestCase8002TempWorkSpace
Ouput, after ESAPI does canonicalization and validation:
C:\UsersTESTUS~1AppDataLocalTempTestCase8002TempWorkSpace
Here is the line of code that causes me to receive the error;
String validatedSourcePath = ESAPI.validator().getValidInput("directoryUnzip", directory, "DirectoryName", 255, false);
File validFile = new File(validatedSourcePath);
#C.Williams: I was about 30 minutes into writing up a detailed reply in an editor and accidentally excited my editor window. I'm too ticked off at my stupidity of not saving it to write it again, especially since I was only about 75% done.
However, if you want to email me I can arrange to talk to you via Google Hangouts or Signal to tell you want your problem is and how you can fix it. But it's long and complicated and partially related to a bug the ESAPI team just fixed but is not in any official release yet. But I am not going to take another 45 minutes or more trying to reply with any written text. My email address should be easy enough to find. Just google for my name and ESAPI. I am one of the project co-leaders on ESAPI.
-kevin wall
I am using JMeter to test HLS playback from a Streaming Server. So, the first HTTP request is for a master manifest file(m3u8). Say,
http://myserver/application1/subpath1/file1.m3u8
The reply to this will result in a playlist something like,
subsubFolder/360p/file1.m3u8
subsubFolder/480p/file1.m3u8
subsubFolder/720p/file1.m3u8
So, next set of URLs become
http://myserver/application1/subpath1/subsubFolder/360p/file1.m3u8
http://myserver/application1/subpath1/subsubFolder/480p/file1.m3u8
http://myserver/application1/subpath1/subsubFolder/720p/file1.m3u8
Now, individual reply to these further will be an index of chunks, like
0/file1.ts
1/file1.ts
2/file2.ts
3/file3.ts
Again, we have next set of URLs as
http://myserver/application1/subpath1/subsubFolder/360p/0/file1.ts
http://myserver/application1/subpath1/subsubFolder/360p/1/file1.ts
http://myserver/application1/subpath1/subsubFolder/360p/2/file1.ts
http://myserver/application1/subpath1/subsubFolder/360p/3/file1.ts
This is just the case of one set(360p). There will be 2 more sets like these(for 480p, 720p).
I hope the requirement statement is clear uptill this.
Now, the problem statement.
Using http://myserver/application1 as static part, regex(.+?).m3u8 is applied at 1st reply which gives subpath1/subsubFolder/360p/file1. This, is then added to the static part again, to get http://myserver/application1/subpath1/subsubFolder/360p/file1 + .m3u8
The problem comes at the next stage. As, you can see, with parts extracted previously, all I'm getting is
http://myserver/application1/subpath1/subsubFolder/360p/file1/0/file1.ts
The problem is obvious, an extra file1, 360p/file1 in place of 360p/0.
Any suggestions, inputs or alternate approaches appreciated.
If I understood the problem correctly, all you need is the file name as the other URLs can be constructed with it. Rather than using http://myserver/application1 as static part of your regex, I would try to get the filename directly:
([^\/.]+)\.m3u8$
# match one or more characters that are not a forward slash or a period
# followed by a period
# followed by the file extension (m3u8)
# anchor the whole match to the end
Now consider your urls, e.g. http://myserver/application1/subpath1/subsubFolder/360p/file1.m3u8, the above regex will capture file1, see a working demo here. Now you can construct the other URLs, e.g. (pseudo code):
http://myserver/application1/subpath1/subsubFolder/360p/ + filename + .m3u8
http://myserver/application1/subpath1/subsubFolder/360p/ + filename + /0/ + filename + .ts
Is this what you were after?
Make sure you use:
(.*?) - as Regular Expression (change plus to asterisk in your regex)
-1 - as Match No.
$1$- as template
See How to Load Test HTTP Live Media Streaming (HLS) with JMeter article for detailed instructions.
If you are ready to pay for a commercial plugin, then there is an easy and much more realistic solution which is a plugin for Apache JMeter provided by UbikLoadPack:
Besides doing this job for you, it will simulate the way a player would read the file. It will also scale much better than any custom script or player solution.
It supports VOD and Live which are quite difficult to script.
See:
http://www.ubik-ingenierie.com/blog/easy-and-realistic-load-testing-of-http-live-streaming-hls-with-apache-jmeter/
http://www.ubik-ingenierie.com/blog/ubikloadpack-http-live-streaming-plugin-jmeter-videostreaming-mpegdash/
Disclaimer, we are the providers of this solution
When I use CRF++ toolkit for the first time. I have installed it and when I run the crflearn command I'am getting this message :
CRF++: Yet Another CRF Tool Kit Copyright (C) 2005-2013 Taku Kudo, All rights reserved. encoder.cpp(340) [feature_index.open(templfile, trainfile)] feature_index.cpp(135) [ifs] open failed: template_file
Can anyone help me?
The package is pretty well documented here
Can you give the full command you tried to use?
The pattern is crf_learn -t [template_file] [data_file_in_the_right_format]
The package comes with an example template file, here's the head of mine:
U00:%x[-5,0]
U01:%x[-4,0]
U02:%x[-3,0]
U03:%x[-2,0]
U04:%x[-1,0]
U05:%x[0,0]
The data needs to be in a tab delimited format with each data point represented by a row, each column is the feature value for that row and the final column is your gold standard label.
Does that help?
You put wrong file path for template featurefile and model. So, put these files inside CRF-PP folder and fire the crf_learn command. Hope this will solve your issue.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 5 years ago.
Improve this question
I have several finished, older PHP projects with a lot of includes that I would like to document in javadoc/phpDocumentor style.
While working through each file manually and being forced to do a code review alongside the documenting would be the best thing, I am, simply out of time constraints, interested in tools to help me automate the task as much as possible.
The tool I am thinking about would ideally have the following features:
Parse a PHP project tree and tell me where there are undocumented files, classes, and functions/methods (i.e. elements missing the appropriate docblock comment)
Provide a method to half-way easily add the missing docblocks by creating the empty structures and, ideally, opening the file in an editor (internal or external I don't care) so I can put in the description.
Optional:
Automatic recognition of parameter types, return values and such. But that's not really required.
The language in question is PHP, though I could imagine that a C/Java tool might be able to handle PHP files after some tweaking.
Thanks for your great input!
I think PHP_Codesniffer can indicate when there is no docblock -- see the examples of reports on this page (quoting one of those) :
--------------------------------------------------------------------------------
FOUND 5 ERROR(S) AND 1 WARNING(S) AFFECTING 5 LINE(S)
--------------------------------------------------------------------------------
2 | ERROR | Missing file doc comment
20 | ERROR | PHP keywords must be lowercase; expected "false" but found
| | "FALSE"
47 | ERROR | Line not indented correctly; expected 4 spaces but found 1
47 | WARNING | Equals sign not aligned with surrounding assignments
51 | ERROR | Missing function doc comment
88 | ERROR | Line not indented correctly; expected 9 spaces but found 6
--------------------------------------------------------------------------------
I suppose you could use PHP_Codesniffer to at least get a list of all files/classes/methods that don't have a documentation; from what I remember, it can generate XML as output, which would be easier to parse using some automated tool -- that could be the first step of some docblock-generator ;-)
Also, if you are using phpDocumentor to generate the documentation, can this one not report errors for missing blocks ?
After a couple of tests, it can -- for instance, running it on a class-file with not much documentation, with the --undocumentedelements option, such as this :
phpdoc --filename MyClass.php --target doc --undocumentedelements
Gives this in the middle of the output :
Reading file /home/squale/developpement/tests/temp/test-phpdoc/MyClass.php -- Parsing file
WARNING in MyClass.php on line 2: Class "MyClass" has no Class-level DocBlock.
WARNING in MyClass.php on line 2: no #package tag was used in a DocBlock for class MyClass
WARNING in MyClass.php on line 5: Method "__construct" has no method-level DocBlock.
WARNING in MyClass.php on line 16: File "/home/squale/developpement/tests/temp/test-phpdoc/MyClass.php" has no page-level DocBlock, use #package in the first DocBlock to create one
done
But, here, too, even if it's useful as a reporting tool, it's not that helpful when it comes to generating the missing docblocks...
Now, I don't know of any tool that will pre-generate the missing docblocks for you : I generally use PHP_Codesniffer and/or phpDocumentor in my continuous integration mecanism, it reports missing docblocks, and, then, each developper adds what is missing, from his IDE...
... Which works pretty fine : there is generally not more than a couple of missing docblocks every day, so the task can be done by hand (and Eclipse PDT provides a feature to pre-generate the docblock for a method, when you are editing a specific file/method).
Appart from that, I don't really know any fully-automated tool to generate docblocks... But I'm pretty sure we could manage to create an interesting tool, using either :
The Reflection API
token_get_all to parse the source of a PHP file.
After a bit more searching, though, I found this blog-post (it's in french -- maybe some people here will be able to understand) : Ajout automatique de Tags phpDoc à l'aide de PHP_Beautifier.
Possible translation of the title : "Automatically adding phpDoc tags, using PHP_Beautifier"
The idea is actually not bad :
The PHP_Beautifier tool is pretty nice and powerful, when it comes to formating some PHP code that's not well formated
I've used it many times for code that I couldn't even read ^^
And it can be extended, using what it calls "filters".
see PHP_Beautifier_Filter for a list of provided filters
The idea that's used in the blog-post I linked to is to :
create a new PHP_Beautifier filter, that will detect the following tokens :
T_CLASS
T_FUNCTION
T_INTERFACE
And add a "draft" doc-block just before them, if there is not already one
To run the tool on some MyClass.php file, I've had to first install PHP_Beautifier :
pear install --alldeps Php_Beautifier-beta
Then, download the filter to the directory I was working in (could have put it in the default directory, of course) :
wget http://fxnion.free.fr/downloads/phpDoc.filter.phpcs
cp phpDoc.filter.phpcs phpDoc.filter.php
And, after that, I created a new beautifier-1.php script (Based on what's proposed in the blog-post I linked to, once again), which will :
Load the content of my MyClass.php file
Instanciate PHP_Beautifier
Add some filters to beautify the code
Add the phpDoc filter we just downloaded
Beautify the source of our file, and echo it to the standard output.
The code of the beautifier-1.php script will like this :
(Once again, the biggest part is a copy-paste from the blog-post ; I only translated the comments, and changed a couple of small things)
require_once 'PHP/Beautifier.php';
// Load the content of my source-file, with missing docblocks
$sourcecode = file_get_contents('MyClass.php');
$oToken = new PHP_Beautifier();
// The phpDoc.filter.php file is not in the default directory,
// but in the "current" one => we need to add it to the list of
// directories that PHP_Beautifier will search in for filters
$oToken->addFilterDirectory(dirname(__FILE__));
// Adding some nice filters, to format the code
$oToken->addFilter('ArrayNested');
$oToken->addFilter('Lowercase');
$oToken->addFilter('IndentStyles', array('style'=>'k&r'));
// Adding the phpDoc filter, asking it to add a license
// at the beginning of the file
$oToken->addFilter('phpDoc', array('license'=>'php'));
// The code is in $sourceCode
// We could also have used the setInputFile method,
// instead of having the code in a variable
$oToken->setInputString($sourcecode);
$oToken->process();
// And here we get the result, all clean !
echo $oToken->get();
Note that I also had to path two small things in phpDoc.filter.php, to avoid a warning and a notice...
The corresponding patch can be downloaded there : http://extern.pascal-martin.fr/so/phpDoc.filter-pmn.patch
Now, if we run that beautifier-1.php script :
$ php ./beautifier-1.php
With a MyClass.php file that initialy contains this code :
class MyClass {
public function __construct($myString, $myInt) {
//
}
/**
* Method with some comment
* #param array $params blah blah
*/
public function doSomething(array $params = array()) {
// ...
}
protected $_myVar;
}
Here's the kind of result we get -- once our file is Beautified :
<?php
/**
*
* PHP version 5
*
* LICENSE: This source file is subject to version 3.0 of the PHP license
* that is available through the world-wide-web at the following URI:
* http://www.php.net/license/3_0.txt. If you did not receive a copy of
* the PHP License and are unable to obtain it through the web, please
* send a note to license#php.net so we can mail you a copy immediately.
* #category PHP
* #package
* #subpackage Filter
* #author FirstName LastName <mail>
* #copyright 2009 FirstName LastName
* #link
* #license http://www.php.net/license/3_0.txt PHP License 3.0
* #version CVS: $Id:$
*/
/**
* #todo Description of class MyClass
* #author
* #version
* #package
* #subpackage
* #category
* #link
*/
class MyClass {
/**
* #todo Description of function __construct
* #param $myString
* #param $myInt
* #return
*/
public function __construct($myString, $myInt) {
//
}
/**
* Method with some comment
* #param array $params blah blah
*/
public function doSomething(array $params = array()) {
// ...
}
protected $_myVar;
}
We can note :
The license block at the beginning of the file
The docblock that's been added on the MyClass class
The docblock that's been added on the __construct method
The docblock on the doSomething was already present in our code : it's not been removed.
There are some #todo tags ^^
Now, it's not perfect, of course :
It doesn't document all the stuff we could want it too
For instance, here, it didn't document the protected $_myVar
It doesn't enhance existing docblocks
And it doesn't open the file in any graphical editor
But that would be much harder, I guess...
But I'm pretty sure that this idea could be used as a starting point to something a lot more interesting :
About the stuff that doesn't get documented : adding new tags that will be recognized should not be too hard
You just have to add them to a list at the beginning of the filter
Enhancing existing docblocks might be harder, I have to admit
A nice thing is this could be fully-automated
Using Eclipse PDT, maybe this could be set as an External Tool, so we can at least launch it from our IDE ?
Since PHPCS was already mentioned, I throw in the Reflection API to check for missing DocBlocks. The article linked below is a short tutorial on how you could approach your problem:
http://www.phpriot.com/articles/reflection-api
There is also a PEAR Package PHP_DocBlockGenerator that can Create the file Page block and the DocBlocks for includes, global variables, functions, parameters, classes, constants, properties and methods (and other things).
php-tracer-weaver can instrument code and generate docblocks with the parameter types, deducted through runtime analysis.
You can use the Code Sniffer for PHP to test your code against a predefined set of coding guidelines. It will also check for missing docblocks and generate a report you can use to identify the files.
The 1.4.x versions of phpDocumentor have the -ue option (--undocumentedelements) [1], which will cause undocumented elements to be listed as warnings on the errors.html page that it generates during its doc run.
Further, PHP_DocBlockGenerator [2] from PEAR looks like it can generate missing docblocks for you.
[1] -- http://manual.phpdoc.org/HTMLSmartyConverter/HandS/phpDocumentor/tutorial_phpDocumentor.howto.pkg.html#using.command-line.undocumentedelements
[2] -- http://pear.php.net/package/PHP_DocBlockGenerator
We use codesniffer for this functionality at work, using standard PEAR or Zend standards. It will not allow you to edit the files on the fly, but will definitely give you a list, with lines and description of what kind of docblock is missing.
HTH,
Jc
No idea if it's any help, but if Codesniffer can point out the functions/methods, then a decent PHP IDE (I offer PHPEd) can easily inspect and scaffold the PHPDoc comments for each function.
Simply type /** above each function and press ENTER, and PHPEd will auto-complete the code with #param1, #param1, #return, etc. filled out correctly, ready for your extra descriptions. Here's the first one I tried in order to provide an example:
/**
* put your comment here...
*
* #param mixed $url
* #param mixed $method
* #param mixed $timeout
* #param mixed $vars
* #param mixed $allow_redirects
* #return mixed
*/
public static function curl_get_file_contents($url, $method = 'get', $timeout = 30, $vars = array(), $allow_redirects = true)
This is easily tweaked to:
/**
* Retrieves a file using the cURL extension
*
* #param string $url
* #param string $method
* #param int $timeout
* #param array $vars parameters to pass to cURL
* #param int $allow_redirects boolean choice to follow any redirects $url serves up
* #return mixed
*/
public static function curl_get_file_contents($url, $method = 'get', $timeout = 30, $vars = array(), $allow_redirects = true)
Not exactly an automated solution, but quick enough for me as a lazy developer :)
You want to actually automate the problem of filling in the "javadoc" type data?
The DMS Software Reengineering Toolkit could be configured to do this.
It parses source text just like compilers do, builds internal compiler structures, lets you implement arbitrary analyses, make modification to those structures, and then regenerate ("prettyprint") the source text changed according to the structure changes. It even preserves comments and formatting of the original text; you can of course insert additional comments and they will appear and this seems to be your primary goal. DMS does this for many languages, including PHP
What you would want to do is parse each PHP file, locate every class/method, generate the "javadoc" comments that should be that entity (difference for classes and methods, right?) and then check that corresponding comments were actually present in the compiler structures. If not, simply insert them. PrettyPrint the final result.
Because it has access to the compiler structures that represent the code, it shouldn't be difficult to generate parameter and return info, as you suggested. What it can't do, of course, is generate comments about intendend purpose; but it could generate a placeholder for you to fill in later.
I had to do a large batch of automation of docblock fixing recently, mostly based on the correct answer above kwith some context-specific changes. It's a hack, but I'm linking back here in case it's useful to anyone else in the future. Essentially, it does basic parsing on comment block tokens within PHP Beautifier.
https://gist.github.com/israelshirk/408f2656100196e73367